Animal health assessment

ABSTRACT

The present teachings generally include techniques for characterizing the health of an animal (e.g., gastrointestinal health) using image analysis (e.g., of a stool sample) as a complement, alternative, or a replacement to biological specimen sequencing. The present teachings may also or instead include techniques for personalizing a health and wellness plan (including, but not limited to, a dietary supplement such as a customized formula based on a health assessment), where such a health and wellness plan may be based on one or more of the health characterization techniques described herein. The present teachings may also or instead include techniques or plans for continuous care for an animal, e.g., by executing health characterization and heath planning techniques in a cyclical fashion. A personalized supplement system (e.g., using a personalized supplement, personalized dosing device, and personalized packaging) may also or instead be created using the present teachings.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/880,836 filed on Jul. 31, 2019, where the entire content of theforegoing application is incorporated by reference herein.

FIELD

The present disclosure generally relates to techniques for assessinganimal health—e.g., through analysis of an image of a biologicalsample—and techniques for creating personalized systems for the animal.

BACKGROUND

Over 70% of an animal's immunity lives in the gastrointestinal tract. Asalient regulator of this immunity is the microbiome—the uniqueenvironment of micro-organisms, both beneficial and harmful, thatinhabit the body of the animal. Because each animal's life is unique,the microbiome of every animal also tends to be quite unique. This posessignificant challenges when assessing animal health and determiningtreatment or other courses of action for the animal. Further, mostanimals do not have advanced means of communicating their wellness andailments to caretakers. Therefore, caretakers are often forced to relyon cues such as diet, appetite, and bowel movements to infer the stateof health of the animal.

Currently, animal health assessment may include submission of a physicalsample of an animal's stool or questionnaire responses related tohealth, behavior, current diet, and other ethnographic information,where this information is then analyzed and compared to a referencedatabase in order to provide a personalized health plan. Thesetechniques can require intricate and complicated steps that often takeseveral weeks (or months) to process, which is often a major barrier tocollecting a reliable sample and diagnosing in sufficient time toprovide a viable solution for a particular animal.

There remains a need for improvements in animal health assessment.

SUMMARY

The present teachings generally include techniques for characterizingthe health of an animal (e.g., gastrointestinal health) using imageanalysis (e.g., of a stool sample) as a complement, alternative, or areplacement to biological specimen sequencing. The present teachings mayalso or instead include techniques for personalizing a health andwellness plan (including, but not limited to, a dietary supplement suchas a customized formula based on a health assessment), where such ahealth and wellness plan may be based on one or more of the healthcharacterization techniques described herein. The present teachings mayalso or instead include techniques or plans for continuous care for ananimal, e.g., by executing health characterization and heath planningtechniques in a cyclical fashion. A personalized supplement system(e.g., using a personalized supplement, personalized dosing device, andpersonalized packaging) may also or instead be created using the presentteachings.

In an aspect, a method of analyzing a stool sample image to provide ahealth assessment of an animal disclosed herein may include: receivingone or more features of a stool sample calculated from one or moreregions of interest within an image including the stool sample;applying, using a model created by identifying a number of associationsbetween one or more image-based features of stool and one or more of amicrobiome characteristic and a metabolome characteristic in stool, theone or more features of the stool sample to the number of associationsin the model to determine a likelihood of a state of one or more of amicrobiome and a metabolome in the stool sample; based on at least thelikelihood of the state of one or more of the microbiome and themetabolome, predicting a health characteristic of an animal thatdeposited the stool sample; and providing a treatment in view of thehealth characteristic.

Implementations may include one or more of the following features. Thetreatment may include a customized health plan for the animal. Thecustomized health plan may include one or more of a behavioral changeand a dietary change. The customized health plan may include arecommendation regarding one or more of diet, sleep, exercise, and anactivity. The treatment may include one or more of a food, a supplement,and a medicine. The treatment may include a personalized dietarysupplement for the animal. The personalized dietary supplement mayinclude a predetermined amount of one or more of a probiotic, aprebiotic, a digestive enzyme, an anti-inflammatory, a natural extract,a vitamin, a mineral, an amino acid, a short-chain fatty acid, an oil,and a formulating agent. One or more features may relate to at least oneof a geometric attribute, a color attribute, and a texture attribute ofthe stool sample. One or more features may relate to a geometricattribute, a color attribute, and a texture attribute of the stoolsample, and the geometric attribute may include one or more of ageometric property and a derived attribute related to geometry, thecolor attribute may include one or more of a color and a derivedattribute related to the color, and the texture attribute may includeone or more of a texture property and a derived attribute related totexture. One or more features may be calculated using a convolutionalneural network (CNN) model. The method may further include: receivingthe image; identifying and extracting one or more regions of interestwithin the image for further analysis, the one or more regions ofinterest including at least a first region of interest having only thestool sample therein; and calculating at least one of a geometricattribute, a color attribute, and a texture attribute of the stoolsample to identify the one or more features of the stool sample. Themethod may further include providing a report for the animal thatincludes the health characteristic. One or more features of the stoolsample may include at least one of a color, a texture, a number ofbinaries, an area, a perimeter, a circularity, a mass, an eccentricity,a major axis, a minor axis, a viscosity, a consistency, a moisturecontent, a solidity, an extent, an equivalent diameter, a specularity, acoherence, a reflectance, a diffusivity, and a presence of a non-stoolsubstance. One or more features of the stool sample may include themass, where the mass is calculated from a geometry and the textureattribute of the stool sample. One of more features of the stool samplemay include the mass, where the mass is calculated from at least one ofa color and a derived color vector of the stool sample. The healthcharacteristic may include a Bristol stool score. The image includingthe stool sample may include a resting surface having markings thereon,the markings including one or more of a known size, a known shape, and aknown color, where the markings are used at least in part forcalculating the one or more features of the stool sample. The method mayfurther include receiving metadata associated with the image, themetadata including a questionnaire response related to one or more of ahealth, a behavior, a current diet, a supplement, a medication,ethnographic information, a breed, a weight of the animal, a weight ofthe stool sample, and a size of the animal, and wherein the metadata isused at least in part in predicting the health characteristic.

In an aspect, a computer program product for analyzing a stool sampleimage to provide a health assessment of an animal disclosed herein mayinclude computer executable code embodied in a non-transitory computerreadable medium that, when executing on one or more computing devices,performs the steps of: receiving one or more features of a stool samplecalculated from one or more regions of interest within an imageincluding the stool sample; applying, using a model created byidentifying a number of associations between one or more image-basedfeatures of stool and one or more of a microbiome characteristic and ametabolome characteristic in stool, the one or more features of thestool sample to the number of associations in the model to determine alikelihood of a state of one or more of a microbiome and a metabolome inthe stool sample; based on at least the likelihood of the state of oneor more of the microbiome and the metabolome, predicting a healthcharacteristic of an animal that deposited the stool sample; andproviding a treatment in view of the health characteristic.

In an aspect, a system for analyzing a stool sample image to provide ahealth assessment of an animal disclosed herein may include a datanetwork, a user device coupled to the data network, and a remotecomputing resource coupled to the data network and accessible to theuser device through the data network, the remote computing resourceincluding a processor and a memory, the memory storing code executableby the processor to perform the steps of: receiving an image including astool sample from the user device over the data network; receiving oneor more features of the stool sample calculated from one or more regionsof interest within the image; applying, using a model created byidentifying a number of associations between one or more image-basedfeatures of stool and one or more of a microbiome characteristic and ametabolome characteristic in stool, the one or more features of thestool sample to the number of associations in the model to determine alikelihood of a state of one or more of a microbiome and a metabolome inthe stool sample; based on at least the likelihood of the state of oneor more of the microbiome and the metabolome, predicting a healthcharacteristic of an animal that deposited the stool sample; andtransmitting a treatment to the user device over the data network inview of the health characteristic.

In an aspect, a method of analyzing a stool sample image to provide ahealth assessment of an animal disclosed herein may include: receivingan image, the image including a stool sample; identifying and extractingone or more regions of interest within the image for further analysis,the one or more regions of interest including at least a first region ofinterest having only the stool sample therein; calculating one or moreof a geometric attribute, a texture attribute, and a color attribute inthe first region of interest to identify one or more features of thestool sample; and applying a model to the one or more features of thestool sample, the model predicting a health characteristic of an animalthat deposited the stool sample.

Implementations may include one or more of the following features. Themethod may further include providing a treatment in view of the healthcharacteristic. The treatment may include a customized health plan forthe animal. The customized health plan may include one or more of abehavioral change and a dietary change. The customized health plan mayinclude a recommendation regarding one or more of diet, sleep, exercise,and an activity. The treatment may include one or more of a food, asupplement, and a medicine. The treatment may include a personalizeddietary supplement for the animal. The personalized dietary supplementmay include a predetermined amount of one or more of a probiotic, aprebiotic, a digestive enzyme, an anti-inflammatory, a natural extract,a vitamin, a mineral, an amino acid, a short-chain fatty acid, an oil,and a formulating agent. The method may further include analyzing theone or more features of the stool sample in view of a reference databaseto determine one or more of the health characteristic or the treatment.The reference database may be a historical database including data fromanalyses of other stool samples. At least one of the other stool samplesmay be from the animal that deposited the stool sample. The other stoolsamples may be from animals distinct from the animal that deposited thestool sample. The method may further include analyzing the healthcharacteristic in view of a reference database to determine thetreatment. The reference database may be a historical database includingdata from analyses of other stool samples. The method may furtherinclude providing a report for the animal that includes the healthcharacteristic. The model may include one or more of a machine-learningmodel and a probabilistic model. The method may further include trainingthe model using the one or more features of the stool sample. The modelmay be part of a recommendation engine configured to provide one or morerecommendations for a treatment in view of the health characteristic.Extracting the one or more regions of interest may include asegmentation of the image performed at least in part by the model. Thehealth characteristic may include a classification on the Bristol stoolscale. The classification may be based at least in part on the colorattribute. The color attribute may be used at least in part fordetermining an attribute related to consistency of the stool sample. Themodel may apply one or more of a weight and a score to the one or morefeatures for predicting the health characteristic. The weight and thescore may be customized for the animal that deposited the stool sample.The method may further include performing microbiome DNA gene sequencingon the stool sample and applying results from the microbiome DNA genesequencing as a factor in predicting the health characteristic of theanimal. The method may further include performing metabolomicssequencing on the stool sample and applying results from themetabolomics sequencing as a factor in predicting the healthcharacteristic of the animal. The method may further include performingfurther analysis including one or more of mass spectroscopy,conductance, and rheology on the stool sample, and using results of thefurther analysis as a factor in predicting the health characteristic ofthe animal. The one or more features of the stool sample may include atleast one of a color, a texture, a number of binaries, an area, aperimeter, a circularity, a mass, an eccentricity, a major axis, a minoraxis, a viscosity, a consistency, a moisture content, a solidity, anextent, an equivalent diameter, a specularity, a coherence, areflectance, a diffusivity, and a presence of a non-stool substance. Theone or more features of the stool sample may include the mass, where themass is calculated from a geometry and the texture attribute of thestool sample. The one of more features of the stool sample may includethe mass, where the mass is calculated from at least one of a color anda derived color vector of the stool sample. The one of more features ofthe stool sample may include the presence of a non-stool substance,where the non-stool substance includes a foreign object. The foreignobject may include one or more of a parasite and a pathogen. Calculatingthe geometric attribute may include: converting the first region ofinterest to grayscale; converting the first region of interest tobinary; and applying one or more morphological operations to the firstregion of interest. Calculating the texture attribute may include use ofa gray level co-occurrence matrix (GLCM). Use of the GLCM may includeplotting a plurality of points for identifying a cluster thereof. Thecolor attribute may be one of a red-green-blue color model, ared-green-blue-alpha color model, a hue-saturation-value color model,and a CIELAB color model. The color attribute may be calculated using amultidimensional color plane. The image may include metadata. Themetadata may include one or more of a time, a date, and a geographiclocation. The method may further include receiving metadata associatedwith the image. The metadata may include a questionnaire responserelated to one or more of a health, a behavior, a current diet, asupplement, a medication, ethnographic information, a breed, a weight ofthe animal, a weight of the stool sample, and a size of the animal. Themetadata may include DNA gene sequencing on the stool sample. Themetadata may include one or more of geolocation information andphysiological information. The metadata may include a ground truthattribute. The ground truth attribute may include one or more of aweight of the stool sample, a Bristol stool score, and a manualsegmentation. The metadata may include historical data. The image may bestored in a remote database. Receiving the image may include retrievingthe image from the remote database. The image may include a backgrounddistinct from the stool sample, where the one or more regions ofinterest include a second region of interest having at least a portionof the background therein. Extracting the one or more regions ofinterest may include identifying the stool sample and the backgroundwithin at least a portion of the image. The method may further includecreating the first region of interest based on an identification of onlythe stool sample, and creating the second region of interest based on anidentification of both a portion of the stool sample and a portion ofthe background. The method may further include classifying the firstregion of interest and the second region of interest for separateanalysis thereof. Extracting the one or more regions of interest mayinclude a manual segmentation of the image. Extracting the one or moreregions of interest may include an automatic segmentation of the image.The automatic segmentation may include utilizing one or more semanticsegmentation models using deep learning. Ground truth data may be usedas an input in the one or more semantic segmentation models to train andvalidate the one or more semantic segmentation models. The one or moresemantic segmentation models may include a u-net network. The one ormore semantic segmentation models may be used with at least one of dataaugmentation, k-folding, and an additional input of data. Extracting theone or more regions of interest may include a combination of a manualsegmentation of the image and an automatic segmentation of the image.The image may include a background distinct from the stool sample. Theone or more regions of interest may include a second region of interesthaving at least a portion of the background therein. The method mayfurther include normalizing one or more of the first region of interestand the second region of interest to account for image variability,thereby creating a normalized image for further standardized analysis.Identifying the one or more features of the stool sample may occur byanalyzing the normalized image. Normalizing the one or more regions ofinterest may include: extracting a color attribute and a dimensionalattribute of the stool sample from the second region of interest;calculating a correction factor for color and aspect ratio using theextracted color attribute and the extracted dimensional attribute; andapplying the correction factor to the first region of interest. Amarking on the background in the second region of interest may be usedfor extracting one or more of the color attribute and the dimensionalattribute. The marking may have one or more of a known size and a knownshape. The marking may include one or more alphanumeric characters. Themarking may include a plurality of colors. The dimensional attribute mayinclude a length. Normalizing the one or more regions of interest mayaccount for one or more image acquisition settings used in capturing theimage. The one or more image acquisition settings may include one ormore of a focal length, a color setting, a lighting setting, and amagnification. Normalizing the one or more regions of interest mayinclude a resizing of the one or more regions of interest. Thebackground may include a resting surface having predetermined markingsthereon.

In an aspect, a computer program product for analyzing a stool sampleimage to provide a health assessment of an animal disclosed herein mayinclude computer executable code embodied in a non-transitory computerreadable medium that, when executing on one or more computing devices,performs the steps of: receiving an image, the image including a stoolsample; identifying and extracting one or more regions of interestwithin the image for further analysis, the one or more regions ofinterest including at least a first region of interest having only thestool sample therein; calculating one or more of a geometric attribute,a texture attribute, and a color attribute in the first region ofinterest to identify one or more features of the stool sample; andapplying a model to the one or more features of the stool sample, themodel predicting a health characteristic of an animal that deposited thestool sample.

In an aspect, a system for analyzing a stool sample image to provide ahealth assessment of an animal disclosed herein may include: a datanetwork; a user device coupled to the data network; and a remotecomputing resource coupled to the data network and accessible to theuser device through the data network, the remote computing resourceincluding a processor and a memory. The memory may store code executableby the processor to perform the steps of: receiving an image from theuser over the data network, the image including a stool sample;identifying and extracting one or more regions of interest within theimage for further analysis, the one or more regions of interestincluding at least a first region of interest having only the stoolsample therein; calculating one or more of a geometric attribute, atexture attribute, and a color attribute in the first region of interestto identify one or more features of the stool sample; and applying amodel to the one or more features of the stool sample, the modelpredicting a health characteristic of an animal that deposited the stoolsample. The code may further perform the step of transmitting atreatment to the user device over the data network in view of the healthcharacteristic.

In an aspect, a method of analyzing an image to provide a healthassessment of an animal as disclosed herein may include: receiving animage, the image including a biological sample of the animal;identifying and extracting one or more regions of interest within theimage for further analysis, the one or more regions of interestincluding at least a first region of interest having only the biologicalsample therein; calculating one or more of a geometric attribute, atexture attribute, and a color attribute in the first region of interestto identify one or more features of the biological sample; and applyinga model to the one or more features of the biological sample, the modelpredicting a health characteristic of an animal from which thebiological sample originated.

Implementations may include one or more of the following features. Thebiological sample may include one or more of skin of the animal, fur ofthe animal, a portion of a mouth of the animal, a portion of an ear ofthe animal, a portion of an eye of the animal, and a portion of a noseof the animal. The method may further include providing a treatment inview of the health characteristic. The treatment may include apersonalized product for the animal. The personalized product mayinclude one or more of a food, a supplement, and a medicine. Thepersonalized product may include one or more of a grooming product, ashampoo, a conditioner, a lotion, a cream, a medicine, an ear drop, aneye drop, a topical substance, a toothpaste, an oral rinse, and a chew.

In an aspect, a method of analyzing a biological sample image to providea health assessment of an animal as disclosed herein may include:receiving one or more features of a biological sample calculated fromone or more regions of interest within an image including the biologicalsample; applying, using a model created by identifying a number ofassociations between one or more image-based features of biologicalsamples and one or more of a microbiome characteristic and a metabolomecharacteristic, the one or more features of the biological sample to thenumber of associations in the model to determine a likelihood of a stateof one or more of a microbiome and a metabolome in the biologicalsample; based on at least the likelihood of the state of one or more ofthe microbiome and the metabolome, predicting a health characteristic ofan animal from which the biological sample originated; and providing atreatment in view of the health characteristic. The biological samplemay include a stool sample.

In an aspect, a method of formulating a personalized product for ananimal disclosed herein may include: receiving an image including abiological sample therein; applying a model to the image to extract oneor more features of the biological sample; and, based at least on theone or more features of the biological sample extracted from the model,selecting one or more ingredients of a personalized product for ananimal from which the biological sample originated.

Implementations may include one or more of the following features. Themethod may further include: combining the one or more ingredients toform the personalized product; packaging the personalized product; anddistributing the personalized product to one or more of the animal and auser associated with the animal. The method may further include dosingthe personalized product for the animal. The personalized product mayinclude a personalized dietary product. The personalized dietary productmay include one or more of a food, a supplement, and a medicine. Thepersonalized dietary product may include a predetermined amount of oneor more of a probiotic, a prebiotic, a digestive enzyme, ananti-inflammatory, a natural extract, a vitamin, a mineral, an aminoacid, a short-chain fatty acid, an oil, and a formulating agent. Thepersonalized product may include one or more of a grooming product, ashampoo, a conditioner, a lotion, a cream, a medicine, an ear drop, aneye drop, a topical substance, a toothpaste, an oral rinse, and a chew.

In an aspect, a personalized product disclosed herein may include one ormore ingredients derived from a computer-based analysis of one or morefeatures of a biological sample extracted from a model applied to animage including the biological sample. The biological sample may bestool, where the personalized product includes a personalized dietaryproduct. The one or more ingredients may include a predetermined amountof one or more of a probiotic, a prebiotic, a digestive enzyme, ananti-inflammatory, a natural extract, a vitamin, a mineral, an aminoacid, a short-chain fatty acid, an oil, and a formulating agent.

These and other features, aspects, and advantages of the presentteachings will become better understood with reference to the followingdescription, examples, and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the devices,systems, and methods described herein will be apparent from thefollowing description of particular embodiments thereof, as illustratedin the accompanying drawings. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating the principles of thedevices, systems, and methods described herein. In the drawings, likereference numerals generally identify corresponding elements.

FIG. 1 illustrates a system for animal health assessment, in accordancewith a representative embodiment.

FIG. 2 is a flow diagram illustrating a technique for generating acustomized health plan for an animal based at least in part on imageanalysis of a biological sample, in accordance with a representativeembodiment.

FIG. 3 is a flow chart of a method for assessing animal health based atleast in part on image analysis of a biological sample, in accordancewith a representative embodiment.

FIG. 4 is a flow chart of a method for dimensionality reduction in thehighly dimensional space of microbiome DNA gene sequencing data, inaccordance with a representative embodiment.

FIG. 5 is a flow diagram illustrating a dynamic recommendation enginewith a feedback loop, in accordance with a representative embodiment.

FIG. 6 is a flow chart of a method for training a model, in accordancewith a representative embodiment.

FIG. 7 is a flow chart of a method for providing a recommendation usinga model, in accordance with a representative embodiment.

FIG. 8 is a flow chart of a method of analyzing an image to provide ahealth assessment of an animal, in accordance with a representativeembodiment.

FIG. 9 is a flow chart of a method of analyzing an image to provide ahealth assessment of an animal, in accordance with a representativeembodiment.

FIG. 10 shows an image and various color planes thereof, in accordancewith a representative embodiment.

FIG. 11 shows an image and segmentation thereof, in accordance with arepresentative embodiment.

FIG. 12 is a flow chart of a method of formulating a personalizedproduct for an animal.

DETAILED DESCRIPTION

The embodiments will now be described more fully hereinafter withreference to the accompanying figures, in which preferred embodimentsare shown. The foregoing may, however, be embodied in many differentforms and should not be construed as limited to the illustratedembodiments set forth herein. Rather, these illustrated embodiments areprovided so that this disclosure will convey the scope to those skilledin the art.

All documents mentioned herein are hereby incorporated by reference intheir entirety. References to items in the singular should be understoodto include items in the plural, and vice versa, unless explicitly statedotherwise or clear from the text. Grammatical conjunctions are intendedto express any and all disjunctive and conjunctive combinations ofconjoined clauses, sentences, words, and the like, unless otherwisestated or clear from the context. Thus, the term “or” should generallybe understood to mean “and/or” and so forth.

Recitation of ranges of values herein are not intended to be limiting,referring instead individually to any and all values falling within therange, unless otherwise indicated herein, and each separate value withinsuch a range is incorporated into the specification as if it wereindividually recited herein. The words “about,” “approximately” or thelike, when accompanying a numerical value, are to be construed asindicating a deviation as would be appreciated by one of ordinary skillin the art to operate satisfactorily for an intended purpose. Similarly,words of approximation such as “about,” “approximately,” or“substantially” when used in reference to physical characteristics,should be understood to contemplate a range of deviations that would beappreciated by one of ordinary skill in the art to operatesatisfactorily for a corresponding use, function, purpose, or the like.Ranges of values and/or numeric values are provided herein as examplesonly, and do not constitute a limitation on the scope of the describedembodiments. Where ranges of values are provided, they are also intendedto include each value within the range as if set forth individually,unless expressly stated to the contrary. The use of any and allexamples, or exemplary language (“e.g.,” “such as,” or the like)provided herein, is intended merely to better illuminate the embodimentsand does not pose a limitation on the scope of the embodiments. Nolanguage in the specification should be construed as indicating anyunclaimed element as essential to the practice of the embodiments.

In the following description, it is understood that terms such as“first,” “second,” “top,” “bottom,” “up,” “down,” and the like, arewords of convenience and are not to be construed as limiting termsunless specifically stated to the contrary.

In general, the devices, systems, and methods disclosed herein generallyrelate to evaluating animal health, including but not limited togastrointestinal health. This may include, but is not limited to,microbiome and/or metabolome assessment, and employing specimen imageassessment as a complement and/or substitute for microbiome and/ormetabolome assessment methods, e.g., traditional microbiome and/ormetabolome assessment methods. The present teachings may also or insteadinclude a personalized health system, including but not limited to apersonalized digestive supplement, cosmetic preparation, dosing method,and package for transporting the health system.

Before detailing the present teachings, some context related to moretraditional health assessment techniques may be helpful. For example,current techniques for assessing gastrointestinal microbiome may includea user providing a stool specimen and/or questionnaire responses relatedto health, behavior, current diet, and other ethnographic information,which are then analyzed and compared to a reference database in order toprovide a personalized health plan. An example of this are websites thatoffer a matching service whereby a user fills out a questionnaire andcan take an optional microbiome assessment in order to achieve the bestmatch for a nutrition plan, including a specific dog food offering.Another example is U.S. Pat. No. 9,633,831, which is hereby incorporatedby reference herein.

Furthermore, stool visual assessment may also include a diagnosticassessment such as that described in U.S. Pat. App. Pub. No.2017/0303901, which is hereby incorporated by reference herein, andwhich appears to contemplate a feces color detection device provided ona toilet seat. However, such a technique may be rather impractical forhigh adoption rates.

An example of personalized health systems and products includes U.S.Pat. App. Pub. No. 2017/0148348, which is hereby incorporated byreference herein, and which appears to contemplate a method forgenerating a personalized supplement recommendation. Moreover, U.S. Pat.App. Pub. No. 2017/0156386, which is hereby incorporated by referenceherein, appears to contemplate a system for producing a nutritionalcomposition where the nutritional dispenser is operatively linked to acontroller and is configured to produce the personalized nutritionalcomposition. Lastly, U.S. Pat. No. 8,762,167, which is herebyincorporated by reference herein, appears to contemplate a system forderiving a personalized health and fitness plan for an individual.Unfortunately, these health systems may require lengthy and involvedanalyses that can often take several weeks to months to process, and dueto the acute nature of some health conditions, this may be anunacceptable process from a patient engagement and a diagnosis/treatmenttimeline perspective.

A personalized supplement typically may be derived from laborious,time-intensive methods, which may not be ideal for animals. For example,U.S. Pat. App. Pub. No. 2011/0014351, which is hereby incorporated byreference herein, appears to contemplate a computer-implemented method,apparatus, system, and packaging for providing a daily nutritionalsupplement component regimen, including a series of daily packets.However, the hurdle of feeding supplements to animals may not be solvedby this technique. Regardless of the ability of a computer to create theappropriate formula, the dosing method should be ideal for intake on aconsistent basis. Administering pills to an animal may be a significantbarrier to compliance.

Thus, existing techniques may not substantially address the ability toprovide relatively fast turnaround times for digestive healthassessments linked to scientific biomarkers as an indicator of health orillness, and a translation of such assessments into a personalizedsolution that enforces compliance. As such, disclosed herein are systemsand methods that can utilize an analysis of an image of a biologicalsample (e.g., a client-supplied image of a stool sample) as a basis forforming a health assessment for an animal related thereto—e.g., inaddition to or instead of traditional laboratory analyses andinformation gathering such as questionnaires and the like.

It will be understood that, although the present teachings may emphasizethe use and analysis of an image of a stool sample, the techniquesdiscussed herein could also or instead be used to analyze images ofother biological material and/or physiological areas. For example, inaddition to or instead of an image of stool, the following is anon-exclusive list of other images that can be analyzed using oradapting the present techniques for providing a health assessment basedon the analyses: urine, vomit, bile, blood, other biological discharge,hair, skin, teeth, gums, tongue, nose, ears, eyes, extremities (e.g.,arms, legs, feet, paws, and so on), nails, throat, anus, sexual and/orreproductive organs, other portions of an animal's body, combinationsthereof, and the like. Furthermore, the present techniques may be usedto provide a health assessment based on an image of an abnormality(e.g., a growth such as a tumor, a rash, an infection, a blemish, and soon), an injury, a mutation, and so on. Thus, the “biological sample” asused herein will be understood to include one or more of the above, andsimilar.

As used herein, it will be understood that the term “animal” as usedherein shall generally refer to any living (or previously living)organism having specialized sense organs and a nervous system, e.g.,where the living organism is able to respond to stimuli. Examples ofanimals include, but are not limited to, companion animals (e.g., dogs,cats, and the like), primates, humans, lizards, and other zoologicalcreatures. For example, an aspect of the present teachings may be usedby a pet owner for obtaining a health assessment of their pet based atleast in part on an analysis of an image of their pet's stool (or animage of another biological feature or deposit of the pet). Inparticular, in an aspect of the present teachings, at least a portion ofthe image analysis, and/or useful information garnered therefrom, can beobtained relatively quickly—e.g., while walking one's dog, a user cantake an image of a stool sample, upload the image to a web-basedplatform, and obtain at least a partial analysis of the image in nearreal time.

The term “personalized” and its conjugates as used herein shallgenerally refer to an outcome that has been customized or tailoredspecifically to an animal or a user.

The term “questionnaire” as used herein shall generally refer to anysolicitation of information from a user (e.g., a caretaker or apatient), which can be recorded electronically, written physically, andso on. A “questionnaire” may also or instead include a solicitation ofinformation from an animal as that term is defined herein. In thismanner, by way of example, a “questionnaire” may include a solicitationof data from sources other than an image submitted for analysis—forexample, geolocation information, heart rate information or otherphysiological information (which may be measured via a wearable deviceor measured manually, e.g., via a stethoscope from a trained healthassociate), diet or supplement information, activity information, age,sex, weight, stool weight, health condition and history, medications,surgeries, species or breed, and the like. Thus, a “questionnaire” mayalso or instead include a solicitation of medical information, e.g.,from a medical exam. Information derived from such a “questionnaire” maythus include any of the foregoing types of information or similar. Itwill also be understood that information derived from such a“questionnaire” may be included as “metadata” as that term is usedherein and defined below, and a questionnaire itself may also beincluded as metadata.

The term “metadata” as used herein in the context of the presentteachings shall generally refer to any data usable in the presentteachings that is different from data generated in an analysis of thecontent of an image—e.g., any data distinct from data derived directlyfrom the image processing techniques described herein. However, it willbe understood that metadata may include information associated with theimage that is different from an image analysis of its content—e.g., adate, a time, a file name, a file type, file directory and/or otherstorage-related information, a file source, a creator of the file, afile size, and the like.

Moreover, as described above, information derived from a questionnairemay be considered metadata. Also or instead, metadata may include datareceived from a laboratory analysis, e.g., a laboratory analysis of abiological sample from an animal or of the animal itself. In general,such data received from a laboratory analysis will be understood toinclude data derived using a quantitative tool and/or data that includesa standardized output. It will thus be further be understood that thehealth assessment techniques included herein may utilize image analysisin combination with metadata such as laboratory-based analyses,including without limitation one or more of a microbiome assessment, ametabolome assessment, bloodwork analysis, urine analysis, a biopsy,parasitology assessment, and so on. In certain embodiments, however,image analysis may be used in lieu of a laboratory-based analysis.Further, in such aspects, the image analysis may be used to makeassessments regarding output that is traditionally reserved for alaboratory-based analyses, such as determining the presence or absenceof certain micro-organisms and/or metabolites based solely on an imageanalysis of a stool sample. And while a focus of the description mayinclude the microbiome and/or metabolome, the use of such image analysisas described herein to predict other health characteristics (e.g.,deficiencies in diet) would be recognized by a person of ordinary skillin the art, and therefore does not depart from the spirit and scope ofthe present teachings.

The term “attribute” as used herein shall generally refer to an imageartifact (e.g., a color channel, an abundance of a substance shown in animage, and the like). This can include one or more of a visible aspector property of content of an image, a measurable aspect or property ofcontent of an image, a calculated aspect or property of content of animage, and/or an aspect or property derived from any of the foregoing orsimilar. Thus, in general, an “attribute” may include something that ismeasurable (e.g., a fundamentally measurable entity), calculatable,and/or otherwise derivable through an image analysis. By way of example,specific attributes that can be useful in the context of the presentteachings may include one or more of a geometric attribute, a textureattribute, and/or a color attribute of contents of an image—where theseattributes may be calculatable or otherwise discernable from analyzingthe image.

The term “feature” as used herein shall generally refer to an input ordatapoint used by a model of the present teachings, e.g., where thefeature is specially tailored for use by the model. In this manner, afeature can include one or more attributes as described above. However,it will be understood that some attributes may not be usable by a modelin their raw, unedited, untranslated, and/or untransformed form, andthus for such attributes to become features, further action may berequired such as one or more of editing, reformatting, transforming, andthe like. The same may be true for metadata—a feature can include somemetadata, and/or metadata can be manipulated or transformed to become afeature to be used as input by a model of the present teachings. In thismanner, a feature may include a derivative and/or the wrangling of a rawattribute (e.g., pixel intensity within an image) and/or metadata (e.g.,a setting of a camera that captured the image) for use by a model.

The term “characteristic” as described herein may include its standarddictionary definition such as a distinguishing quality of an item(animate or inanimate). As such, a characteristic as used herein mayhave some overlap with an attribute as described herein and definedabove, but while an attribute is generally measured, calculated, and/orotherwise derived from an image, a characteristic as used herein cangenerally include qualities that can be measured, calculated, and/orotherwise derived (or may otherwise exist or be known) independent fromany image or image analysis.

A high-level workstream of an aspect of the present teachings will nowbe described for context and by way of example. First, a user uploads adigital image (e.g., photograph) of a biological sample from ananimal—e.g., a digital photograph of a stool sample from their pet thatis taken with the user's smartphone and uploaded via a mobileapplication, electronic mail, and/or a website—to a database managementsystem, which then prepares the image for analysis. Images may thus beacquired from users and retrieved from database management systems andthe like. The image may then be passed into an image analysis pipeline,where this pipeline may include analyses operations such as one or moreof region of interest extraction, image normalization, featureengineering, and a modeling phase. One or more models may ultimatelypredict one or more health indices based on one or more features derivedfrom the image.

FIG. 1 illustrates a system for animal health assessment, in accordancewith a representative embodiment. More particularly, the system 100 maybe used for analyzing a stool sample image to provide a healthassessment of an animal. In general, the system 100 may include anetworked environment where a data network 102 interconnects a pluralityof participating devices and/or users 101 in a communicatingrelationship. The participating devices may, for example, include anynumber of user devices 110, remote computing resources 120, and otherresources 130. Generally, the system 100 may be used for any of theimplementations of the present teachings described herein. For example,the system 100 may be used for analyzing an image 112 of a biologicalsample 103 to provide a health assessment 150 of an animal 104. Morespecifically, in the system 100, a user 101 may capture or otherwiseretrieve an image 103 of the biological sample 103 related to the animal104, transmit that image 103 over the data network 102 to a remotecomputing resource 120 for processing and analysis, where the remotecomputing resource 120 then provides output of the analysis (e.g., ahealth assessment 150, which may be in the form of a report or the like)to the user 101 over the data network 102. This entire process can bedone relatively quickly, e.g., in near real-time (such as less than fiveminutes, less than one minute, or mere seconds). Certain participantsand aspects of the system 100 will now be described.

The user 101 may be associated with the animal 104 and the user device110. For example, the animal 104 may include a pet (e.g., a dog, a cat,and the like), and the user 101 may be an owner of the pet, a member ofthe same household as the pet, or otherwise associated with the pet(e.g., a caretaker, medical personnel, and the like). In some instances,the user 101 themselves may be the animal 104—i.e., where the animal 104is a human. The user 101 may also or instead include a medicalprofessional (e.g., a doctor, a veterinarian, a nurse, and the like), aresearcher, a scientist, a laboratory technician, a student, and so on.In some instances, the user 101 may not be human, but instead the user101 may include a computing device, computer program, or the like—e.g.,where the user 101 is a computer-program product comprising computerexecutable code embodied in a non-transitory computer readable mediumthat, when executing on one or more computing devices (e.g., the userdevice 110)—that is configured to capture, create, edit, receive, and/ortransmit an image 112 for processing and analysis as described hereinfor obtaining a health assessment 150 or the like.

The animal 104 may be any as described herein—i.e., any living (orpreviously living) organism having specialized sense organs and anervous system. In certain implementations, the animal 104 is acompanion animal such as a dog, a cat, and the like.

The biological sample 103 may be related to the animal 104. For example,in an aspect, the biological sample 103 includes stool that was excretedby the animal 104. In an aspect where the biological sample 103 includesstool that was excreted by the animal 104, the biological sample 103 mayalso or instead include other substances deposited by the animal 104 (inaddition to, or instead of, stool) such as a foreign object (e.g., partof a chew toy), blood, a parasite (e.g., a tapeworm and/or itseggs/larvae), hair, microorganisms, microbiomes, metabolomes, and thelike. Thus, the biological sample 103 may include stool and one or moreother substances in an aspect.

The biological sample 103 may also or instead include a portion of theanimal 104 itself. For example, the biological sample 103 may include aphysiological area (internal and/or external) of the animal 104. In thismanner, the biological sample 103 may include at least a portion of oneor more of skin of the animal 104, fur of the animal 104 (and/or hair ofthe animal 104), a mouth of the animal 104 (e.g., one or more of teeth,gums, plaque, tongue, throat passage, and the like), an ear of theanimal 104, an eye of the animal 104, a nose of the animal 104, thethroat of the animal 104, an extremity of the animal 104 (e.g., arms,legs, feet, paws, and so on), nails of the animal 104, the anus of theanimal 104, sexual and/or reproductive organs of the animal 104, anorgan of the animal 104, blood vessels of the animal 104, a muscle ofthe animal 104, a joint of the animal 104, a tendon of the animal 104,other portions of an animal's body, and so on. Furthermore, thebiological sample 103 may also or instead include an abnormality of theanimal 104 (e.g., a growth such as a tumor, a rash, an infection, ablemish, and so on), an injury, a mutation, and so on.

The biological sample 103 may also or instead include other secretions,discharges, internal substances, and so on. For example, the biologicalsample 103 may include urine, vomit, bile, blood, other biologicaldischarge, and the like.

Thus, as discussed above, it will be understood that, although thepresent teachings may emphasize the use and analysis of an image 112 ofa stool sample, the system 100 may also or instead include otherbiological samples 103 including other biological material and/orphysiological areas.

The data network 102 may be any network(s) or internetwork(s) suitablefor communicating data and information among participants in the system100. This may include public networks such as the Internet, privatenetworks, telecommunications networks such as the Public SwitchedTelephone Network or cellular networks using third generation (e.g., 3Gor IMT-2000), fourth generation (e.g., LTE (E-UTRA) or WiMAX-Advanced(IEEE 802.16m)), fifth generation (e.g., 5G), and/or other technologies,as well as any of a variety of corporate area or local area networks andother switches, routers, hubs, gateways, and the like that might be usedto carry data among participants in the system 100.

Each of the participants of the data network 102 may include a suitablenetwork interface comprising, e.g., a network interface card, which termis used broadly herein to include any hardware (along with software,firmware, or the like to control operation of same) suitable forestablishing and maintaining wired and/or wireless communications. Thenetwork interface card may include without limitation a wired Ethernetnetwork interface card (“NIC”), a wireless 802.11 networking card, awireless 802.11 USB device, or other hardware for wired or wirelesslocal area networking. The network interface may also or instead includecellular network hardware, wide-area wireless network hardware or anyother hardware for centralized, ad hoc, peer-to-peer, or other radiocommunications that might be used to connect to a network and carrydata. In another aspect, the network interface may include a serial orUSB port to directly connect to a local computing device such as adesktop computer that, in turn, provides more general networkconnectivity to the data network 102.

The user devices 110 may include any devices within the system 100operated by one or more users 101 for practicing the techniques ascontemplated herein. The user devices 110 may thus be coupled to thedata network 102. Specifically, the user devices 110 may include anydevice for capturing an image 112 (e.g., a photograph)—or otherwisecreating, preparing, editing, or receiving the image 112—andtransmitting the image 112 for analysis (e.g., over the data network102). To this end, the user device 110 may include a camera 114 or thelike, or the user device 110 may otherwise be in communication with acamera 114 or the like. In a preferred implementation, the user device110 includes a smartphone or the like having an internal camera 114,processing capability, and access to the data network 102, all in a onedevice. The user device 110 may also or instead include any device forreceiving output of an analysis of the image 112 over the data network102, e.g., displaying such output on a graphical user interface 116thereof. Similarly, the user device 110 may include any device forcreating, preparing, editing, receiving, and/or transmitting (e.g., overthe data network 102) other data or files in the system 100, such asmetadata 118 related to the image 112, the animal 104, the user 101, andso on. The user devices 110 may also or instead include any device formanaging, monitoring, or otherwise interacting with tools, platforms,and devices included in the systems and techniques contemplated herein.The user devices 110 may be coupled to the data network 102, e.g., forinteraction with one or more other participants in the system 100. Itwill also be understood that all or part of the functionality of thesystem 100 described herein may be performed on the user device 110 (oranother component of the system 100) without a connection to the datanetwork 102—by way of example, a closed network native application on asmartphone may be utilized, whereby functionality (e.g., one or more ofthe models 128 described herein) can run in a closed environment.

By way of further example, the user devices 110 may include one or moredesktop computers, laptop computers, network computers, tablets, mobiledevices, portable digital assistants, messaging devices, cellularphones, smartphones, portable media or entertainment devices, or anyother computing devices that can participate in the system 100 ascontemplated herein. As discussed above, the user devices 110 mayinclude any form of mobile device, such as any wireless, battery-powereddevice, that might be used to interact with the networked system 100. Itwill also be appreciated that one of the user devices 110 may coordinaterelated functions (e.g., performing processing and/or analysis of theimage 112 and the like) as they are performed by another entity such asone of the remote computing resources 120 or other resources 130.

Each user device 110 may generally provide a user interface 116. Theuser interface 116 may be maintained by a locally-executing applicationon one of the user devices 110 that receives data from, for example, theremote computing resources 120 or other resources 130. In otherembodiments, the user interface 116 may be remotely served and presentedon one of the user devices 110, such as where a remote computingresource 120 or other resource 130 includes a web server that providesinformation through one or more web pages or the like that can bedisplayed within a web browser or similar client executing on one of theuser devices 110. The user interface 116 may in general create asuitable visual presentation for user interaction on a display device ofone of the user devices 110, and provide for receiving any suitable formof user input including, e.g., input from a keyboard, mouse, touchpad,touch screen, hand gesture, or other use input device(s).

The remote computing resources 120 may include, or otherwise be incommunication with, a processor 122 and a memory 124, where the memory124 stores code executable by the processor 122 to perform varioustechniques of the present teachings. More specifically, a remotecomputing resource 120 may be coupled to the data network 102 andaccessible to the user device 110 through the data network 102, wherethe remote computing resource 120 includes a processor 122 and a memory124, where the memory 124 stores code executable by the processor 122 toperform the steps of a method according to the present teachings—such asany of the methods or techniques described herein.

The remote computing resources 120 may also or instead include datastorage, a network interface, and/or other processing circuitry. In thefollowing description, where the functions or configuration of a remotecomputing resource 120 are described, this is intended to includecorresponding functions or configuration (e.g., by programming) of aprocessor 122 of the remote computing resource 120, or in communicationwith the remote computing resource 120. In general, the remote computingresources 120 (or processors 122 thereof or in communication therewith)may perform a variety of processing tasks related to analyzing an image112 of a biological sample 103 to provide a health assessment of ananimal 104 related to the biological sample 103 as discussed herein. Forexample, the remote computing resources 120 may manage informationreceived from one or more of the user devices 110 (e.g., the image 112,metadata 118, and so on), and provide related supporting functions suchas parsing or segmentation of the image 112 for analysis, normalizationof the image 112, performing calculations, identifying and extractingvarious properties and attributes of the image 112, calculating featuresof the contents of the image 112, applying one or more models 128 and/oralgorithms to the image 112 and/or metadata 118, retrieving and/oranalyzing information from a database 140 and/or the memory 124,providing a health assessment 150, communicating with other resources130 and the participants in the system 100, storing data, and the like.The remote computing resources 120 may also or instead include backendalgorithms that react to actions performed by a user 101 at one or moreof the user devices 110. These backend algorithms may also or instead belocated elsewhere in the system 100.

The remote computing resources 120 may also or instead include a webserver or similar front end that facilitates web-based access by theuser devices 110 to the capabilities of the remote computing resource120 or other components of the system 100. A remote computing resource120 may also or instead communicate with other resources 130 in order toobtain information for providing to a user 101 through a user interface116 on the user device 110. Where the user 101 specifies certaincriteria for analysis or otherwise, this information may be used by aremote computing resource 120 (and any associated algorithms) to accessother resources 130. Additional processing may be usefully performed inthis context such as recommending certain analyses and processingoperations and techniques.

A remote computing resource 120 may also or instead maintain, orotherwise be in communication with, a database 140 of data 142, andoptionally with an interface for users 101 at the user devices 110 toutilize the data 142 of such a database 140. Thus, in one aspect, aremote computing resource 120 may include a database 140 of data 142,and the remote computing resource 120 may act as a server that providesa platform for selecting and using such data 142, and/or providingsupporting services related thereto. The database 140 may be a localdatabase of the remote computing resource 120, or a remote database tothe remote computing resource 120 or another participant in the system100. Thus, the database 140 may include a cloud-based database or thelike.

A remote computing resource 120 may also or instead be configured tomanage access to certain content (e.g., for a particular user 101). Inone aspect, a remote computing resource 120 may manage access to acomponent of the system 100 by a user device 110 according to input froma user 101.

Thus, and as described throughout the present disclosure, a remotecomputing resource 120 coupled to the data network 102 and accessible tothe user device 110 through the data network 102 may include a processor122 and a memory 124, where the memory 124 stores code executable by theprocessor 122 to perform the steps of: receiving an image 112 from theuser 101 over the data network 102, the image 112 including a biologicalsample 103 which may be in the form of a stool sample; identifying andextracting one or more regions of interest within the image 112 forfurther analysis, the one or more regions of interest including at leasta first region of interest having only the stool sample therein;calculating one or more attributes, including but not limited to ageometric attribute, a texture attribute, and/or a color attribute inthe first region of interest to identify one or more features of thestool sample; and applying a model 128 to one or more features of thestool sample, the model 128 predicting a health characteristic 152 of ananimal 104 that deposited the stool sample. The code may further performthe step of transmitting a treatment 154 to the user device 110 over thedata network 102 in view of the health characteristic 152, e.g., in areport or another form of a health assessment 150.

The data 142 stored in a database 140 of the system 100 may includereference information for use by the remote computing resource 120 forproviding the health assessment 150. For example, this data 142 mayinclude historical data such as information from analyses of one or morebiological samples 103 (e.g., from the same animal 104 or a differentanimal). The data 142 may also or instead include one or more models128, e.g., for retrieval and use by the remote computing resource 120 oranother participant for processing and analyzing the image 112 or otherinformation to generate the health assessment 150. The data 142 may alsoor instead include a plurality of images 112 (e.g., of the same ordifferent biological sample 103, from the same animal 104 or a differentanimal, and so on). The data 142 may also or instead include a number ofcorrelations and/or associations between one or more image-basedfeatures of a biological sample 103 and one or more of a microbiomepresence and/or characteristic, a metabolome presence and/orcharacteristic, a health characteristic, a diet characteristic, and thelike.

As discussed herein, the systems 100 and techniques of the presentteachings may include and utilize one or more models 128 that areconfigured and programmed to perform certain tasks to assist with thevarious analyses described herein. By way of example, parsing the image112 that is received or retrieved for analysis may involve segmentingthe image 112, which can be performed at least in part by a specificsegmentation model, e.g., a deep learning model. This segmentation modelmay read-in the image 112 and label specific classes in the image 112(e.g., a class for the biological sample 103 and a class forbackground), where the segmentation model can be rewarded for correctlyidentifying certain pixels and punished when the segmentation model isincorrect when identifying certain pixels. The segmentation model maythus be configured to extract the background from the image 112 foranalysis of only the biological sample 103 contained within the image112. The segmentation model may also or instead normalize one or moreattributes of the content within the image 112, normalize one or morecolor planes of the content within the image 112, account for lightingor other conditions, and so on.

A model 128 can also or instead include a geometric model—e.g., a model128 specifically configured and programmed to identify and determine(e.g., calculate) geometric features of content within the image112—e.g., geometric features including but not limited to morphologicalregion properties, which can include one or more of contiguous pixelareas, a perimeter, major/minor axes, and the like. Such a geometricmodel may include machine-learning models such as random forest. Thesemodels can optionally be trained with optimized hyper parameters using agrid search routine, which may have advantageous accuracy. Such ageometric model may also or instead include other machine learningmodels, including without limitation, one or more of k-nearest neighbor,support vector machine, logistic regression, decision trees (which canbe gradient-boosted and/or combined in ensemble architectures), NaïveBayes, Multi-Layer Perceptron, or the like.

A model 128 can also or instead include a color model—e.g., a model 128specifically configured and programmed to identify and determine colorof contents within the image 112, or other color-related attributesand/or features. Such a color model may include a multilinear model,e.g., with lasso correction or the like. The output of color model mayinclude a color index, which can be in the form of a color wheel, a listof colors within a portion of the image 112, a sum of the top colorplanes, and the like. The output of color model may also or insteadinclude a prediction for a Bristol stool score or the like, which can beused as a health characteristic 152 for formulating a treatment 154and/or a health assessment 150.

Also, or instead, k-mean image segmentation may be utilized in thepresent teachings, where a pseudo-color plane is created (e.g.,specifically for the present teachings), which can allow the presentteachings to identify and determine the top ‘n’ most abundant colors inan image. This can also or instead permit the present teachings toidentify and determine color homogeneity within an image.

Supervised machine learning may find an association between data (e.g.,feature vectors X) and a corresponding label (y, which can becategorical or continuous) so that the computer can learn an algorithm,f, that maps the input to the output (e.g., y=f(X)). Two furthersubgroups can include classification and regression problems, where thesupervised machine-learning model is trained to predict categorical data(e.g., predicting Bristol stool score from an image) and continuousdata, respectively (e.g., predicting a mass from an image). Someexamples of models include: support vector machines, stochastic gradientdescent, k nearest neighbors, decision trees, neural networks, and soon, where a longer list can be found here by way of example:https://scikit-learn.org/stable/supervisedlearning.html#supervised-learning.

Unsupervised machine learning may assume a similar structure tosupervised machine learning, except that no training labels y may beused. These models may attempt to learn the underlying structure ordistribution of the data to learn more about its behavior. Some exampleof tasks here are clustering (e.g., principal component analysis (PCA)for microbiome work) and associations (e.g., Apriori algorithms forassociation rule learning). A longer list can be found here by way ofexample: https://scikit-learn.org/stable/unsupervised_learning.html.

Semi-supervised approaches may occur when practitioners feed in apartial list of labeled training data. This typically increases accuracyas a result of using labeled data, but allows for practitioners tominimize cost (e.g., time and monetary to gather labeled data). A longerlist can be found here by way of example:https://scikit-learn.org/stable/modules/label_propagation.html.

Transfer learning may include a technique that leverages open-sourcemodels that have been trained on significantly more images. For example,Inception was trained on 1.4M images, and is a well-known network forclassification tasks. The model and/or model weights can be adapted fora custom task by appending a custom deep neural network after anappropriate layer in Inception to tune the model weights towards acustom classification task according to the present teachings.

In general, models 128 may include one or more of a computer visionmodel (e.g., where such a computer vision model uses semanticsegmentation to detect a region of interest within the image 112), aU-Net segmentation model, a machine-learning model (e.g., to predicthealth of the animal 104 by geometric features and color features of thebiological sample 103), a transfer learning model (e.g., where weightsare adapted from a network trained in an existing model, such as aVGG-16 model—i.e., trained on classifying ten million images), acorrelation model, a deep learning model, and so on.

The other resources 130 may include any resources that can be usefullyemployed in the devices, systems, and methods as described herein. Forexample, these other resources 130 may include without limitation otherdata networks, human actors (e.g., programmers, researchers, annotators,editors, analysts, and so forth), sensors (e.g., audio or visualsensors), data mining tools, computational tools, data monitoring tools,algorithms, and so forth. The other resources 130 may also or insteadinclude any other software or hardware resources that may be usefullyemployed in the networked applications as contemplated herein. Forexample, the other resources 130 may include payment processing serversor platforms used to authorize payment for access, content, oroption/feature purchases, or otherwise. In another aspect, the otherresources 130 may include certificate servers or other securityresources for third-party verification of identity, encryption ordecryption of data, and so forth. In another aspect, the other resources130 may include a desktop computer or the like co-located (e.g., on thesame local area network with, or directly coupled to through a serial orUSB cable) with one of the user devices 110 or remote computingresources 120. In this case, the other resource 110 may providesupplemental functions for the user device 110 and/or remote computingresource 120. Other resources 130 may also or instead includesupplemental resources such as cameras, scanners, printers, inputdevices, and so forth.

The other resources 130 may also or instead include one or more webservers that provide web-based access to and from any of the otherparticipants in the system 100. While depicted as a separate networkentity, it will be readily appreciated that the other resources 130(e.g., a web server) may also or instead be logically and/or physicallyassociated with one of the other devices described herein, and may, forexample, include or provide a user interface for web access to a remotecomputing resource 120 or a database 140 in a manner that permits userinteraction through the data network 102, e.g., from a user device 110.

It will be understood that the participants in the system 100 mayinclude any hardware or software to perform various functions asdescribed herein. For example, one or more of the user device 110 andthe other resources 130 may include a memory 124 and a processor 122.

The various components of the networked system 100 described above maybe arranged and configured to support the techniques, processes, andmethods described herein in a variety of ways. For example, in oneaspect, a user device 110 connects through the data network 102 to aserver (e.g., that is part of one or more of the remote computingresource 120 or other resources 130) that performs a variety ofprocessing tasks related to analyzing an image 112 to provide a healthassessment 150 of an animal 104 associated with a biological sample 103present in the image 112. For example, the remote computing resource 120may include a server that hosts a website (and/or a mobile applicationor application programming interface) that runs a platform for analyzingand/or processing an image 112 and other data. More specifically, a user101 associated with the user device 110 and having appropriatepermissions for using the system 100 may use the user device 110 totransmit an image 112 and/or metadata 118 over the data network 102 tothe remote computing resource 120. The remote computing resource 120 mayreceive the image 112 from the user 101 over the data network 102 forprocessing and analysis thereof, where an output of the processing andanalysis may include a health assessment 150.

The health assessment 150 may include a prediction regarding a healthcharacteristic 152 of the animal 104. Such a health characteristic 152may include one or more of a classification on the Bristol stool scale(i.e., a standard scale designed to classify feces into certain groupsfor a health determination), a weight, whether the animal 104 is likelysick or healthy, a dietary insight, and so on. The health assessment 150may also or instead include a treatment 154 prescription and/orrecommendation for the animal 104. Such a treatment 154 recommendationmay include one or more of a customized health plan, a food, asupplement (e.g., a personalized dietary supplement), a medicine, and soon. The health assessment 150 may be presented or transmitted to theuser 101 in the form of a report or the like. It will be understoodthat, in general and unless stated otherwise, the “report” as describedherein may include any share-out of a result of one or more of theanalyses performed in the present teachings.

By way of example, as part of the health assessment 150 and/or treatment154, techniques may involve the creation of a personalized product 156for the animal 104, and thus the system 100 may include a personalizedproduct 156 and/or related components such as packaging, containers,dosing instruments, and so on. The personalized product 156 may includeone or more ingredients derived from a computer-based analysis of one ormore features of a biological sample 103 extracted from a model 128applied to an image 112 that includes the biological sample 103. Asdescribed herein, the biological sample 103 may include stool, and, insuch instances (and/or in other instances when the biological sample isnot stool) the personalized product 156 may include a personalizeddietary product. In this manner, one or more ingredients of thepersonalized product 156 may include a predetermined amount of one ormore of a probiotic, a prebiotic, a digestive enzyme, ananti-inflammatory, a natural extract, a vitamin, a mineral, an aminoacid, a short-chain fatty acid, an oil, a formulating agent, and thelike. The personalized product 156 may also or instead include any asotherwise described herein.

In an aspect, many of the techniques of the present teachings areperformed by the remote computing resource 120. For example, the remotecomputing resource 120 may include an analysis engine (or otherwise aprocessor 122) configured by computer-executable code to analyze theimage 112 and metadata 118, a recommendation engine (or otherwise aprocessor 122) configured by computer-executable code to provide arecommendation for the animal 104, and so on. However, it will beunderstood that some of the features and functionality described withreference to the remote computing resource 120 may also or instead beperformed by another participant in the system 100.

FIG. 2 is a flow diagram illustrating a technique for generating acustomized health plan for an animal based at least in part on imageanalysis of a biological sample, in accordance with a representativeembodiment. As shown in the figure, inputs 201 may include, but are notlimited to, a biological sample 203 from an animal, an image 212 of thebiological sample 203, and metadata 218. By way of example, these inputs201 can include a biological sample 203 in the form of a stool sample,an image 212 of the stool sample in the form of a digital file, andmetadata 218 in the form of laboratory data and/or questionnaireresponses, which may be related to health, behavior, current diet, otherethnographic information, and the like.

The respective inputs 201 may then be analyzed as shown in the analysis202 portion of the figure. The analysis 202 may include a laboratoryanalysis 211 (e.g., metabolome and/or microbiome analysis, which mayinclude the extraction, amplification, sequencing, and subsequentanalyses thereof), an image analysis 213 of the supplied image 212,and/or a data analysis 202 of the metadata 218 and the like. For eachinput 201, analyzed data may be compared to a reference database.

The outputs 250 may include, but are not limited to, a predictionregarding a health characteristic 252 of the animal, a report 251 whereresults are shared with an end user, and a treatment 254. For example,one or more of the health characteristic 252 and the report 251 may beused to create a treatment 254 such as a customized health plan, whichmay include but is not limited to including a recommendation ofpersonalized dietary supplements along with recommendations for otherfacets of health such as diet, sleep, exercise, activity, and otherlifestyle recommendations for the animal. Thus, the outputs 250 mayinclude shared results of analyses/assessments and personalizedsolutions for the animal that are created from such results.

It will be understood that the technique shown with respect to FIG. 2may or may not include the step of providing the actual biologicalsample 203 for a laboratory analysis 211. Thus, the inputs 201 maysolely include the image 212 and metadata 218 for image analysis 213 anddata analysis 215.

FIG. 3 is a flow chart of a method for assessing animal health based atleast in part on image analysis of a biological sample, in accordancewith a representative embodiment. As shown in step 302, the method 300may include receiving an image of a biological sample, e.g., a raw imageof a stool sample in the form of a digital photo file. In this method300, an image of a stool sample may be analyzed among a collection ofpredetermined factors, including but not limited to color, texture,viscosity, consistency, volume, mass, and the like. The stool sample maybe collected in a controlled manner, e.g., such that visual referencestandards are available in the image itself. This may be achieved byusing a stool sample collection bag with a known color, shape, and/orother markings such as text.

As shown in step 304, the method 300 may include extracting features ofthe biological sample included in the image, and as shown in step 306,the method 300 may include scoring and/or indexing these features. Incertain aspects, based on the raw image comparison to a referencedatabase, a score or index may be assigned for each feature that isextracted. Each score, in isolation and/or aggregate, may providesalient data correlated to the presence or absence of specific symptomsor physiological conditions, which can serve as input into apersonalization algorithm, with findings provided in a report. Thus, asshown in step 308, the method 300 may include generating output such asa report. Such a report may include or be used to create a personalizedhealth plan, including but not limited to a recommendation ofpersonalized dietary supplements along with other facets of healthincluding diet, sleep, exercise, activity, and other lifestylerecommendations.

The features that may be extracted in the method 300 may include, butare not limited to, one or more of color, texture, viscosity,consistency, volume, mass, and the like. To this end, the scores thatmay be assigned can include one or more of a color score, a texturescore, a viscosity score, a consistency score, and a mass score.However, it will be understood that examples of features that areextracted from an image as disclosed herein may be provided herein byway of convenience and simplicity.

FIG. 4 is a flow chart of a method for dimensionality reduction in thehighly dimensional space of microbiome DNA gene sequencing data, inaccordance with a representative embodiment. Specifically, FIG. 4illustrates a method 400 of generating a biome label wheredimensionality is reduced in the highly dimensional space of microbiomeDNA gene sequencing data. As shown in step 402, the method 400 mayinclude providing DNA sequencing data, where this DNA sequencing data isprovided in N dimensional space. As shown in step 404, the method 400may include performing a particular analysis, such as a PrincipalComponent Analysis (PCA) or Non-Negative Matrix Factorization (NMF), toensure non-negative concentrations or relative abundance of, forexample, micro-organisms at the phylum, genus, and/or species taxonomiclevel and/or other taxonomic levels that are physically realizable,which may be performed to reduce the number of dimensional vectors nsuch that n<N. Thus, as shown in step 406, the method 400 may includereducing the dimensional vectors n such that n<N. As shown in step 408,further clustering may be performed to determine biome archetypes (i.e.,under the hypothesis that there is a finite number of known types ofbiome expressions for simplicity) referred to as biome labels. Thus, asshown in step 410, the method 400 may include providing biome labels asan output.

FIG. 5 is a flow diagram illustrating a dynamic recommendation enginewith a feedback loop, in accordance with a representative embodiment.More specifically, FIG. 5 shows a system 500 illustrating a dynamicrecommendation engine 505 with a feedback loop 507 for furtherrefinement, which may be generated to obviate a need for an actualbiological sampling and for ensuring that a personalized/customizedrecommendation 509 is assigned to a specific biome label 511. In otherwords, once the biome labels 511 are identified, the system 500illustrates a recommendation engine 505 that can be formed and appliedto bypass or obviate the need for laboratory analysis of a biologicalsample—i.e., in some aspects, the recommendation engine 505 of thesystem 500, with training, can operate without any need for laboratorydata as input, and merely an image as input.

The recommendation engine 505 may be formed by taking metadata 518 suchas symptomatic data or the like (e.g., from questionnaires), and aninitial recommendation 503 (e.g., a hypothesis based at least in part onthe metadata 518) to deliver a custom recommendation 509 for an animal.The recommendation engine 505 may be dynamically updated and optimizedas more data is added—including but not limited to longitudinal data—tooptimize the correspondence between the biome label 511 and a customrecommendation 509 for an animal.

FIG. 6 is a flow chart of a method for training a model, in accordancewith a representative embodiment. Specifically, the model trained inFIG. 6 may be one or more of an association model (e.g., amachine-learning model that learns associations between one or moreimage-based features identified from an analysis of an image of abiological sample and one or more health characteristics for forming ahealth assessment and/or treatment plan) and/or a correlation model(e.g., a model that correlates one or more image-based featuresidentified from an analysis of an image of a biological sample with abiological label or characteristic, such as likely presence or absenceof particular micro-organisms—it will be understood that a Pearsoncorrelation coefficient may be used to this end). Thus, the method 600may generally be used for developing new feature models, and, morespecifically, the method 600 may include a technique for identifying newfeatures that can be used for machine-learning models to predict healthcharacteristics, including metabolomic and microbiome characteristicsidentified from an analysis of an image of a biological sample with abiological label or feature, or a health characteristic more generally.That is, the method 600 may involve the use of new computer-vision-basedfeatures that can connect images to a likely biological characteristic,such as the presence or absence of a particular micro-organism. It willbe understood that, in some aspects, these are not correlation models,but rather feature vectors to be used in future machine-learning modeldevelopment.

By way of example, the method 600 may involve capturing associations toidentify what features a system can have resolution towards and thesefeatures can be used to build a machine-learning model—e.g., featuressuch as equivalent diameter, which can represent a geometric regionproperty that shows a relatively high correlation with the presence ofthe Bacteroidetes phylum of a host animal's microbiome. In this manner,the present teachings can include the creation of a linear model usingequivalent diameter to predict Bacteroidetes levels.

As discussed above, in certain aspects, the method 600 may include atraining task to establish a biome model (e.g., a canine biome model)further used for testing and/or generating health recommendations,predictions, and/or conclusions. As another useful example, the method600 may include a technique for training a model that correlates afeature extracted from an analysis of an image of a stool sample withthe likely presence or absence of a particular metabolome. It will beunderstood that other correlations and/or associations between featuresextracted from an image and particular health and/or biologicalcharacteristics exist, and that the techniques described herein shallinclude the training and use of models based on these correlationsand/or associations.

As shown in step 602, the method 600 may include acquiring an image of abiological sample, which, for the sake of example, may include a digitalphotograph of a stool sample. This step 602 may thus includespecifically tailoring received images such that the method 600 includesthe use of a substantially consistent procedure in processing images. Tothis end, received images may be processed such that they are in aspecific format with specific characteristics after receipt, and/or thecapture of an image (e.g., via a user's digital camera on a web-enableddevice) may be controlled in a manner that enables substantiallyconsistent processing of such images. Controlling the capture of animage may include adjusting settings for capture and/or providing ameans for substantially consistent image capture, such as by applying abackground for a biological sample that includes predetermined markings,colors, shapes, text, and the like.

As shown in step 604, the method 600 may include acquiring image data.Such image data may include metadata accompanying the image (orotherwise received from a user or the like). Such image data may also orinstead be extracted from the image itself.

As shown in step 606, the method 600 may include processing the image.This may include a signal/image processing analysis or the like toidentify and/or generate image-based features of the biological sample(e.g., stool) such as color, texture, apparent viscosity, approximatevolume and/or mass, consistency, and so on. Thus, in this manner, and asshown in step 608, the method 600 may include extracting features of theimage. Signal/image processing may also or instead include linearfiltering with labeling. To this end, a filter bank or the like may beutilized, where the filter bank is a set of filters (e.g., Gabor filterstuned at various frequencies and phases) and can be located locally inan analysis engine or the like, and/or may be retrieved for use from aremote resource.

As shown in step 610, the method 600 may include receiving data, whichcan include metadata as described herein and/or one or more ofground-truth data, laboratory data, and so on. By way of example, thisstep 610 may include receiving biome data, e.g., a biome label frommicrobiome DNA gene sequencing. Thus, this step 610 may includeanalyzing an actual (i.e., non-image) biological sample (e.g., stool)for certain microbiome/metabolome presence and/or absence, and/or acharacteristic related thereto. Using this data and the featureextractions from image processing, correlations and/or associations maybe uncovered. And, as shown in step 612, the method 600 may includeidentifying these correlations and/or associations (e.g., correlationsbetween the presence and/or absence of certain image features with thepresence and/or absence of certain micro-organisms). Further processingof these parameters could employ a deep learning network—potentially aConvolutional Neural Network (CNN)—as a model to create a biome modelthat learns to associate a biome label with an input image. Otherprocessing can include dimensionality reduction for clustering (e.g.,PCA, Non-Negative Matrix Factorization (NMF), and the like), which canbe further used to analyze correlations and/or associations with imagefeatures. In this manner, the method 600 can involve associating amicrobiome/metabolome characteristic with an image artifact to create anew feature.

Thus, as shown in step 614, the method 600 may include training such amodel. Thus, a feature can be defined by the association between theappearance of a biological sample (e.g., stool) and the presence orabsence of biological content or substances (e.g., microbiomecharacteristics and/or metabolome characteristics). And, as describedabove, other features may also or instead be defined by the appearanceof a biological sample as processed in an image, besides or in additionto the presence or absence of biological substances, such as a healthcharacteristic. One example is that texture attributes of stool can beattributed a level of hydration of an animal that deposited the stool.Further, because more granular features extracted from advanced imageprocessing can yield more granular health characteristics, the presenttechniques can include the development of such features and models.

FIG. 7 is a flow chart of a method for providing a recommendation usinga model, in accordance with a representative embodiment. The method 700may be similar to that shown and described above with reference to FIG.6 , but where the model is employed for making a prediction (andsubsequently a customized, personalized recommendation for an animalbased on a correlation and/or association) without receipt of alaboratory analysis of a physical biological sample from the animal.Thus, as shown in step 702, the method 700 may include acquiring animage of a biological sample (e.g., stool); as shown in step 704, themethod 700 may include acquiring image data; as shown in step 706, themethod 700 may include processing the image; and, as shown in step 708,the method 700 may include extracting one or more attributes and/orfeatures from the image.

As shown in step 710, the method 700 may include applying a trainedmodel that can identify a relationship between an image feature andanother characteristic, such as the presence or absence of a microbiomeand/or metabolome characteristic.

As shown in step 712, the method 700 may include determining a healthcharacteristic based on the correlation and/or association. For example,this can include determining a biome label (and/or a metabolome label,or another label) based on a correlation and/or association to theextracted image feature. And, as shown in step 714, the method 700 mayinclude providing a recommendation based on the determined healthcharacteristic—e.g., a personalized recommendation for the animal.

The method 700 may also or instead include a testing task, e.g., wheresome data is held back from the model for testing the model. Images maybe acquired and processed in a similar manner as for the training taskin FIG. 6 , only that the canine biome model may already be trained andcan now be employed for estimating the learned biome label, and furtheremployed by the recommendation engine to generate a personalizedrecommendation.

Surprisingly, the image analysis algorithm output can be associated orotherwise connected to other input data, including but not limited tostool microbiome DNA sequencing. This non-obvious result may benefit themethod of obviating the stool specimen as a viable alternate means ofdelivering a personalized health insight, which can be embodied in apersonalized health report in a relatively fast manner. Thus, anon-obvious, advantageous result of the present teachings may involve aconnection between image attributes/features to health characteristics(e.g., micro-organism presence/absence and/or other characteristic,Bristol stool score, and so on)—which can be accomplished withhigh-accuracy and substantially faster than a traditional laboratoryanalysis. For example, a traditional analysis of a stool sample mayinvolve requesting a specimen kit and receiving the same (which can takeabout 5 days), collecting the stool sample (which can be burdensome,prone to contamination, and unsanitary depending on the collectiontechnique), mailing the sample to a laboratory (which can take a fewdays), and waiting for the results of extraction, amplification,sequencing, and analysis (which can take about 42 days)—thus, thistraditional analysis can take about 53-55 days in total. In contrast,using an aspect of the present teachings involving image analysis of abiological sample such as a stool sample, a user may capture a photo ofstool at their convenience using a smartphone or the like and upload thephoto (or otherwise transmit a digital photo file) for analysis, wherethe computer-implemented analysis is performed and data is ready for theuser within seconds—the entire process can take less than 1 minute,e.g., 30 seconds or less, 15 seconds or less, or even 5 seconds or less.In an embodiment, the user may repeat the technique outlined above at alater time to attempt to optimize a preexisting health plan and/orpreexisting personalized dietary supplement and/or measure/quantifyhealth outcomes as a function of the new health plan.

FIG. 8 is a flow chart of a method of analyzing an image to provide ahealth assessment of an animal, in accordance with a representativeembodiment. The method 800 may be performed using any of the systems andtechniques described herein, alone or in combination, e.g., the system100 of FIG. 1 . By way of example, the method 800 may include analyzingan image of a stool sample to provide a health assessment of the animalthat excreted the stool sample. However, and similar to other aspects ofthe present disclosure, it will be further understood that while themethod 800 may emphasize the use of an image of a stool sample foranalysis, the method 800 may be employed using an image of another formof a biological sample, such as any as described herein—e.g., one ormore of skin of the animal, fur of the animal, a portion of a mouth ofthe animal, a portion of an ear of the animal, a portion of an eye ofthe animal, a portion of a nose of the animal, and so on. Regardless, ingeneral, the method 800 may include the submission of a photo of abiological sample (e.g., a digital file containing the photo), theprocessing and analyzing of that photo, and providing useful feedbackrelated to health in view of that analysis.

As shown in step 802, the method 800 may include receiving an image,where the image includes a biological sample of an animal. For example,the image may include a digital photograph of a stool sample that wasexcreted by the animal. In this manner, in a preferred embodiment, theimage includes a photo from a smartphone or other web-enabled device(e.g., a digital photograph) that is transmitted to a platform foranalysis (e.g., transmitted over a data network to a remote computingresource for processing and analysis). However, other forms of receivingan image are also or instead possible. For example, the image may bestored in a database, where receiving the image may include retrievingthe image from the database, whether local or remote to a computingresource that performs processing of the image.

It will be understood that, in addition to or instead of receiving animage, the method 800 (or an aspect of the present teachings moregenerally) may include receiving one or more of a video, a plurality ofimages, a three-dimensional scan, and the like. For example, in thiscontext, a video can simply be thought of as a collection of stillimages, where one or more of these images may be analyzed as describedherein. This can yield more input data for an aspect of the presentteachings, and thus more-refined results/recommendations. For example, avideo or the like can provide more angles, views, colors, features, andthe like of a biological sample for image-based analysis. Thus, asdescribed herein, an image being analyzed will be understood aspotentially including one or more of a video, a three-dimensional image,and the like, or a portion thereof—e.g., a still image or other portionof a video, scan, or the like.

As shown in step 804, the method 800 may include receiving metadata,e.g., metadata associated with the image, a user, the animal, orotherwise. In some implementations, the image itself includesmetadata—e.g., a file containing the image, a file name, a filedirectory and/or directory structure, and the like. For example, in thismanner, the metadata may include one or more of a time, a date, ageographic location or other geolocation information, other metadatacommonly associated with a digital photograph or computer filegenerally, and the like. The metadata may also or instead include otherinformation such as a questionnaire response related to one or moreaspects of the animal—e.g., a health, a behavior, a current diet, asupplement, a medication, ethnographic information, a breed, a weight, asize, other physiological information, and the like. The metadata mayalso or instead include laboratory data or the like. For example, themetadata may include bloodwork analysis, DNA gene sequencing on theanimal, results of a microbiome analysis for the animal's stool, resultsof a metabolome analysis for the animal's stool, and the like. Themetadata may also or instead include historical data—e.g., data fromprevious analyses of a biological sample of the animal or analysesrelated to another animal. This historical data may include raw dataand/or derived data.

The metadata may also or instead include information that can be used totest and/or reinforce some of the model analysis techniques of themethod 800. For example, the metadata may include a ground truthattribute such as a weight of the biological sample and/or a Bristolstool score when the biological sample is a stool sample. Such a groundtruth attribute may also or instead include manually segmented images inthe semantic segmentation algorithm.

As shown in step 806, the method 800 may include identifying andextracting one or more regions of interest within the image for furtheranalysis. These regions of interest may include at least a first regionof interest having only the biological sample therein. For example, inan aspect where the biological sample is a stool sample, one or more ofthe regions of interest may include at least a first region of interesthaving only the stool sample therein. When this is the case—i.e., whenonly the biological sample is present within the region of interest andthe region of interest can be identified and extracted forprocessing—normalization of the region of interest may be unnecessary.However, an aspect of the method 800 may include normalization asexplained in more detail below with reference to step 808.

As described herein, in certain aspects, the image includes a backgrounddistinct from the biological sample, and thus one or more regions ofinterest may include a second region of interest having at least aportion of the background therein. In this manner, extracting one ormore regions of interest may include identifying the biological sampleand the background within at least a portion of the image. The method800 may further include creating the first region of interest based onan identification of only the biological sample, and creating the secondregion of interest based on an identification of both a portion of thebiological sample and a portion of the background. Also, or instead, themethod 800 may include classifying the first region of interest and thesecond region of interest for separate analysis thereof.

In certain aspects, the images that are received or otherwise retrievedfor systems and techniques of the present teachings can be highlyvariable. For example, lighting and background conditions can varysignificantly from image to image. For example, lighting and backgroundconditions can vary significantly from image to image. By way ofexample, simple image analysis (e.g., Otsu's threshold and segmentation)may not work well due to the presence of highly variable backgrounds,which is why a U-Net network may be preferred—i.e., a version of asemantic segmentation network that is often used in biomedical imagingapplications. Convolutional neural networks may require certainpre-processing as a way to standardize the images and/or regions thereoffor analysis. Thus, in an aspect, images may be resized and/ornormalized for consistency in inputs where a U-Net network is used. Alsoor instead, a model used herein for such segmentation may be improvedwith data augmentation, k-folding, more data, other semanticsegmentation networks and training patterns, manually segmented images,combinations thereof, and the like.

Extracting a region of interest may thus include segmentation. Forexample, extracting one or more regions of interest may include asegmentation of the image performed at least in part by a model—e.g., asegmentation model or a portion of a model configured to perform morecomprehensive analysis. One or more of these models may include a deeplearning model. In this manner, in an aspect, extracting one or moreregions of interest may include an automatic segmentation of the image,where such an automatic segmentation includes utilizing one or moresemantic segmentation models using deep learning. For example, asemantic segmentation model used herein may include a U-Net network.Also or instead, a semantic segmentation model used herein may be usedwith at least one of data augmentation, k-folding, and an additionalinput of data. Ground truth data may be used as an input in a semanticsegmentation model to train and validate the model. Thus, semanticsegmentation network models can be trained internally (e.g., U-Net)and/or adapted from public models using transfer learning to performimage segmentation. And ground truth data (e.g., with all classeslabeled) can be provided as inputs to help train/validate the model foroptimal accuracy.

Extracting a region of interest may also or instead include a manualsegmentation of the image. In this manner, in an aspect, extracting oneor more regions of interest includes a combination of a manualsegmentation of the image and an automatic segmentation of the image.For example, images may be manually segmented when the dataset isrelatively small. Manual segmentation may help improve a signal to noiseratio (SNR) of features extracted, since regions of interest shouldtheoretically be well-defined. Regardless, in an aspect, automatic ROIsegmentation is performed using semantic segmentation models using deeplearning, which can provide high-quality, autonomous region-of-interestsegmentation.

The output from the extraction of one or more regions of interest mayinclude regions of interest that are classified as one of biologicalsample only, background only, or a combination of biological sample andbackground.

As shown in step 808, the method 800 may include normalizing one or moreregions of interest of the image. Images may be normalized to controlfor different image acquisition settings (e.g., focal lengths,magnification, and the like). Also or instead, color and length featuresof the biological sample may be extracted and/or normalized, e.g., usingmarkings on a known background as correction factors to apply to certainregions of interest. For example, the image may include a backgroundthat is distinct from the biological sample, where normalization isperformed to analyze only the biological sample in a useful manner.Thus, one or more of the regions of interest may include a second regionof interest having at least a portion of the background therein. By wayof example, in an aspect where the biological sample includes a stoolsample, the background can include one or more of a surface (e.g., aman-made surface such as concrete, or a natural surface such as grass)and/or the background can include a bag, container, or the like thatholds or otherwise contains the stool sample. Additionally oralternatively, and as explained in more detail below, the background mayinclude predetermined markings or features that can aid in one or moreof normalization, extracting and identifying a region of interest,extracting and identifying features of a biological sample, and so on.In some aspects, the background can include several types ofbackgrounds, e.g., a combination of one or more of a surfacespecifically tailored for image analysis, grass, dirt, debris, concreteand/or pavement, and the like.

In this manner, the method 800 may include normalizing one or more of afirst region of interest including only the biological sample and asecond region of interest including some background to account for imagevariability, thereby creating a normalized image for furtherstandardized analysis. Identifying one or more features of thebiological sample as explained herein may thus occur by analyzing thisnormalized image.

In an aspect, normalizing one or more regions of interest includesextracting normalization factors, such as a color attribute and adimensional attribute (e.g., a length) of the biological sample from thesecond region of interest that includes at least a portion of thebiological sample and at least a portion of a background. As mentionedabove, the background may be strategically designed to assist in thenormalization process, or otherwise for analysis of the biologicalsample—e.g., in an aspect, the background includes a resting surfacehaving predetermined markings thereon. For example, the background mayinclude a sheet (e.g., a plastic sheet, which can be in the form of abag for collecting and/or disposing of stool) or another surface withmarkings that can be used for normalization. Thus, a marking on thebackground in the second region of interest may be used for extractingone or more of the color attribute and the dimensional attribute. Themarking may include symbols, designs, or text having known attributes.In this manner, the marking may have one or more of a known size and aknown shape (e.g., rectangles, triangles, circles, lines, waves, or thelike, having known dimensions). The marking may also or instead includeone or more alphanumeric characters—e.g., letters and/or numbers. By wayof example, if the markings on a background within an image includecharacters with known sizes, an aspect ratio of the characters may beused to calculate dimensions of contents within the image. Use of theaspect ratio may be applied from an image analysis perspective, e.g.,where the method 800 may include identifying how many pixels correspondto a set distance so that techniques can convert from raw calculationsin pixels to a more useful form. The marking may also or instead includeone or more colors, e.g., a plurality of colors. In this manner, thecolors of the markings can provide a baseline reference for determiningcolors within the biological sample, which can assist in accounting forimage acquisition differences (e.g., one or more of lighting, focus,magnification, and the like).

Normalizing one or more regions of interest may further includecalculating a correction factor for a color and an aspect ratio usingthe extracted color attribute and the extracted dimensional attribute,and applying the correction factor to the first region of interest.Normalizing one or more regions of interest may also or instead includea resizing of a region of interest.

Normalizing one or more regions of interest may account for one or moreimage acquisition settings used in capturing the image, e.g., settingsof a camera application on a user's smartphone or the like. Such animage acquisition setting may include one or more of a focal length, acolor setting, a lighting setting, a magnification, and the like.

As shown in step 810, the method 800 may include calculating one or moreattributes, including but not limited to a geometric attribute, atexture attribute, and/or a color attribute in the first region ofinterest to identify one or more features of the biological sample. Forexample, in an aspect, the method 800 may include calculating each of ageometric attribute, a texture attribute, and a color attribute in thefirst region of interest to identify one or more features of thebiological sample.

The features that are identified in this step 810 may include at leastone of a color, a texture, a number of binaries, an area (e.g., thenumber of pixels that the sample occupies), a perimeter (e.g., theperimeter of the sample that approximates the contour as a line throughthe centers of border pixels using an 8-connectivity, a 4-connectivity,and the like), a circularity (e.g., a measure of how similar the shapeof the sample is to a circle), a volume, a mass, an eccentricity (e.g.,a measure of an aspect ratio computed to be the eccentricity of theellipse that has the same second moments as the sample region), a majoraxis (e.g., the length of the major axis of the ellipse that has thesame normalized second central moments as the sample), a minor axis(e.g., the length of the minor axis of the ellipse that has the samenormalized second central moments as the sample), a minor-major-axisratio (e.g., a measure of the aspect ratio of these axes, meaning theratio of the minor to major axes of the ellipse that has the same secondmoments as the sample region), a viscosity, a consistency, a moisturecontent, a solidity (e.g., a measure of the convexity computed as theratio of the number of pixels in the sample to that of its convex hull),an extent (e.g., a ratio of an area of the sample to its axis-alignedbounding box), an equivalent diameter (e.g., the diameter of a circlewith the same area as the sample), a specularity, a coherence, areflectance, a diffusivity, a presence of substance that is distinctfrom the biological sample (e.g., a non-stool substance when thebiological sample is a stool sample, such as a foreign object orbiological matter), and the like. It will be understood that thefeatures that are identified in this step 810 may include derivedfeatures. By way of example, viscosity may be modeled and then derivedprior to feature engineering.

For example, mass may be identified in step 810. In an aspect, the massis calculated from a geometry and a texture attribute of the biologicalsample. Also or instead, the mass may be calculated from at least one ofa color and a derived color vector of the stool sample. In an aspect, amass model may be implemented in the method 800, where such a model usesfeatures including but not limited to color and geometry attributevectors (x) and ground truth weights from metadata as labels (y). In anaspect, the same test train split procedure may be used to train themodel on about 80% of the data and to evaluate the model's performancewith the remaining about 20%. With the training data, supervised machinelearning regression algorithms (e.g., support vector machines, k-nearestneighbors, random forest regressor, and the like) may be used to predictthe continuous output (which can be chosen to get a weight outputinstead of categorical Bristol stool score labels or the like). Thesemodels may then be optimized using hyperparameter search routines suchas one or more of ‘gridsearch,’ ‘randomsearch,’ and the like.

By way of further example, and as mentioned above, where the biologicalsample is a stool sample, one or more features of the stool sample mayinclude the presence of the non-stool substance. In this manner, thenon-stool substance may include a foreign object—e.g., one or more of aparasite, a pathogen, an inanimate object such as a toy, and the like.

As discussed above, the method 800 may include calculating a colorattribute in a region of interest (e.g., the first region of interest)to identify one or more features of the biological sample. By way ofexample, in an aspect, an output of the method 800 may include a healthcharacteristic in the form of a classification on the Bristol stoolscale, where this classification is based at least in part on the colorattribute. The color attribute may also or instead be used at least inpart for determining an attribute related to consistency, which may beparticularly applicable in the context of a stool sample.

The color attribute may be derived from or otherwise utilize one or moreof a red-green-blue color model, a red-green-blue-alpha color model, ahue-saturation-value color model, and/or a CIELAB color model. Also orinstead, the color attribute may be calculated using a multidimensionalcolor plane. Such a calculated-multidimensional color plane may beprovided using an algorithm such as a k-means algorithm. By way ofexample, the color attribute may be calculated using a dimensionalityreduction approach (e.g., k-means clustering) to cluster regions ofsimilar colors. For example, segmenting ten color planes within a stoolROI may allow techniques to look at color homogeneity within a sample(e.g., color plane 1 may have 30% abundance and be the dominant colorcluster). These homogeneity values can be further used as features fortraining several machine-learning models.

As discussed above, the method 800 may include calculating a geometricattribute in a region of interest (e.g., the first region of interest)to identify one or more features of the biological sample. In an aspect,calculating the geometric attribute includes converting the first regionof interest to grayscale, converting the first region of interest tobinary, and applying one or more morphological operations to the firstregion of interest.

As discussed above, the method 800 may include calculating a textureattribute in a region of interest (e.g., the first region of interest)to identify one or more features of the biological sample. In certainimplementations, calculating the texture attribute includes use of agray level co-occurrence matrix (GLCM), where GLCM is a statisticalmeasure that measures spatial relationships between patterns ofcontiguous pixels. Use of the GLCM may include plotting a plurality ofpoints for identifying a cluster thereof. Thus, it will be understoodthat one or more features of the biological sample that are identifiedthrough calculation of a property from the image may not be humanvisual, and the texture attribute is one example where GLCM is used toextract this attribute. By way of further example, texture can also orinstead be calculated using a Gabor filter bank, which includespermutations of different magnitude, orientation, and frequencies ofGabor wavelets. Each Gabor wavelet may be convolved over an image andmay create a filtered image, with high signal captured from texture ororiented pixels in phase with the Gabor filter. And these filteredimages can be used for the classification of texture. The following is apublicly available script that shows how this sequence can be adaptedand executed in the context of the presentteachings—https://scikit-image.org/docs/stable/auto_examples/features_detection/plot_gabor.html—andone skilled in the art would understand how to implement such atechnique given the present disclosure.

Thus, many feature sets can be calculated from the images, where threeexample feature sets include color, texture, and region properties,which can be calculated at respective points in an analysis pipeline.These feature sets can be used for training a machine learning model. Anexample analysis pipeline to identify these features may includeconverting an image to grayscale, and calculating texture propertiestherefrom. That grayscale image may then be converted to binary, andhave its signal cleaned with morphological operations (e.g.,erosions/dilations) for calculating region properties. The image mayhave global color vectors calculated to get the color attribute andrelated features. In this manner, output of this step 810 may includeone or more of a color, a texture, a number of binaries, an area, acircularity, an eccentricity, a major and minor axis, a perimeter, andthe like.

As shown in step 812, the method 800 may include analyzing one or morefeatures of the biological sample—e.g., the features identified fromcalculating one or more of the geometric attribute, the textureattribute, and the color attribute. For example, this step 812 mayinclude analyzing a feature of the biological sample in view of areference database to determine a health characteristic of the animal ora treatment for the animal. The reference database may be a historicaldatabase that includes data from analyses of other biologicalsamples—e.g., biological samples from the same animal or biologicalsamples from different animals.

As shown in step 814, the method 800 may include applying a model to oneor more features of the biological sample. In general, the model may bepart of a recommendation engine configured to provide one or morerecommendations for a treatment in view of a health characteristicdetermined or identified in the analysis of the image of the biologicalsample. That is, the model may be configured to (e.g., programmed to)predict a health characteristic of the animal from which the biologicalsample originated (e.g., where the biological sample is a stool sample,the model may predict a health characteristic of the animal thatdeposited/excreted the stool sample). It will be understood that, in thecontext of the present teachings, the “model” may include output by oneor more algorithms that can be comprised of model data and a predictionalgorithm. Thus, the algorithm may be thought of as the programming, andthe models may be thought of as representing the program. In thismanner, the models discussed herein in the context of the presentteachings may represent what was learned by one or more machine-learningalgorithms. The model may thus represent output that is saved afterrunning a machine-learning algorithm on training data and can representthe rules, numbers, and any other algorithm-specific data structuresrequired to make predictions. Thus, the model may include both data anda procedure (as set forth by one or more algorithms) for using the datato make a prediction. Therefore, in the context of the presentteachings, a model may learn patterns in the input data (e.g., color,geometry, texture, and the like) and the model may abstract thesepatterns into relationships (e.g., mathematical functions), such that,when new data is entered, the “model” uses these functions it haslearned with the training data to create a prediction (e.g., a healthcharacteristic prediction for formulating a treatment plan).

The model may include a machine-learning model. In general, themachine-learning model may be the output of one or more machine learningalgorithms running on input data—e.g., input data in the form of anidentified features of the biological sample from the processing of theimage of the sample as described herein. The machine-learning model mayutilize an algorithm configured to perform pattern recognition, such asdescribed above.

The model may include a probabilistic model, e.g., a stochastic model orthe like. Probabilistic models may yield a stochastic chance for aninstance to occur. By way of example, this can include the temporaltracking of health data (similar to a time series analysis) where models(e.g., recurrent neural network such as long short-term memory (LSTM))are used to forecast future chances of a health outcome.

The model may apply one or more of a weight and a score to a feature ofthe biological sample for predicting the health characteristic. Theweight and/or the score may be customized for the animal that isassociated with the biological sample. It will be understood that theseweights and/or scores may be tailored for use by a specific model,and/or they can be tailored for use by more than one model (e.g., wheremodels are interconnected).

It will thus be understood that one or more of the models used hereinmay be interconnected within another one or more models. For example,several machine-learning models can contribute towards a healthprediction (e.g., a consistency model, a color model, and so on). In anaspect, the aforementioned weights can provide for an expected averageof each individual model's predictions. Thus, the weights describedherein may include one or more of individual model weights (e.g., acolor model may be a linear regression—y=mx+b—where the weights are ‘m’and ‘b’ values) and weights of an expected average as just described(e.g., health index=a*consistency prediction+b*color prediction). Statedotherwise, a model used in the present teachings may be made up ofseveral models (e.g., a comprehensive model may include one or more of acolor model, a consistency model, and so on), where model weights mayassist in computing an expected average, whereas the model weights in anindividual model (e.g., the color model) may be used fortraining/executing that particular model.

Therefore, it will also be understood that one or more models may beinput to another one or more models. For example, a color model and aconsistency model may each make a prediction for a Bristol stool scorein an aspect. In this example, the expected average of these two valuesmay be used to create the health index score. And this predicted Bristolstool score value may then be provided as an input to a mass model topredict stool mass in grams or weight in ounces.

In this manner, the present teachings may include a system of models,which refers to a combination and/or a collection of individual models,e.g., to make a determination on a derived metric, such as “overallhealth score” and the like. Thus, it will be understood that the modelsmay include individual models and/or ensemble models, e.g., where, inensemble models, the strengths of each model are considered to provide afinal derived outcome. That is, it will be understood that a system ofmodels can be generated using individual models as components (e.g.,inputs to the system of models). For example, an individual supportvector machine model for geometric attributes and a separate individuallinear regression model for color attributes can be defined, where eachmodel can be trained to predict a health outcome, such as Bristol stoolscore. A system of these two models can be defined such that bothindividual models contribute together to predict a health outcome, suchas Bristol stool score. Moreover, an additional system of models can becreated when the output of one individual model (e.g., Bristol stoolscore prediction of a geometric model) is used as a feature for aseparate model (e.g., a mass prediction model).

An aspect of the present teachings may also or instead include trainingone or more models, e.g., for use in step 814 of the method 800. Incertain aspects, training the model includes using one or more featuresof the stool sample that were identified in the image analysis. A fewexamples of training the model in the context of the biological sampleincluding stool are included below, but it will be understood that othertechniques may also or instead be used to train the model(s) herein.

Training the model may begin with data wrangling to prepare for actualtraining. For example, in the context of the biological sample includingstool, various stool health features may be retrieved or otherwisegathered. This may include extracting data from a source, including aspreadsheet, a relational and/or non-relational database (stored locallyor remotely, e.g., on the cloud), and the like. This may include loadingthe data into software, and transforming data as needed (e.g., scalingdata to unit mean and variance, and the like). Data can continue to becleaned prior to analysis (e.g., via one or more of string formatting,removing null values, converting object types, and the like). This mayfurther include formatting feature matrix X and label vector Y, wherefeature matrix X can have any number of features, including regionproperties, color, and so on, and where feature vector y contains a trueclass label (e.g., Bristol stool score).

Preparation of data for training may also include semantic segmentation.As such, ground truth labels may first be created, where these can becreated manually. Next, images and corresponding ground truth labels maybe stored in a data source (e.g., local or remote, e.g., on acloud-based storage system). Images may be extracted from a source(e.g., local or cloud-based storage), resized to standard sizes, andaugmented. Such augmentation may include one or more of cropping,flipping vertically/horizontally, shearing, rotating, and the like. Thiscan be done to increase the dataset size to both raw and ground truthimages. It will be understood that the preparation of data for trainingcould be automated with software.

Continuing with this training example, the modeling will now bediscussed. Training the model may include training to identify stoolhealth features and the like. To this end, the train test may be splitto train the model—e.g., use a portion of data (˜70-80%) to train themodel; reserve some data (˜20-30%) to test the data after training; andfor validating (˜10%) the model during training. The model may becreated, e.g., loaded from a package or a custom defined architecture.The model may then be trained using training data. For this, one canoptionally feed in validation data to validate during training. Modelaccuracy may be evaluated with testing data to compare predictions toground truth data. Model hyperparameters can be optimized using routineslike cross-validation, grid search, and so on, to propose the bestparameters leading to highest accuracy, lowest bias, and/or lowestvariance. The model can be saved to be called at runtime, where themodel may have an application programming interface (API) to be calledprogrammatically at runtime.

Training the model may also or instead include sematic segmentationtraining, which may follow a similar procedure as that described above.Thus, one may define a custom convolutional neural network (CNN)architecture or instantiate CNN architecture (e.g., U-Net). The data setmay be split to train the model—e.g., use a portion of data (˜70-80%) totrain the model; reserve some data (˜20-30%) to test the data aftertraining; and for validating (˜10%) the model during training. The modelmay then be trained using training data. For this, one can optionallyfeed in validation data to validate during training—e.g., evaluate modelaccuracy with testing data comparing predictions to ground truth (e.g.,DICE coefficients). Model hyperparameters can be optimized usingroutines like cross-validation, grid search, and so on, to propose thebest parameters leading to highest accuracy, lowest bias, and/or lowestvariance. The model can be saved to be called at runtime, where themodel may have an API to be called programmatically at runtime.

Training the model may also or instead include training a model toidentify a feature in stool. To this end, pre-trained, open source,model weights (e.g., VGG-16) may be loaded, which may have beenpreviously trained on several million images. The weights may then beadapted for the context of the present teachings.

Image class metadata may be passed in, which can be organized by foldername, and/or passed as a dictionary/list object with corresponding classnames per image. The train test may be split to train the model—e.g.,use a portion of data (˜70-80%) to train the model; reserve some data(˜20-30%) to test the data after training; and for validating (˜10%) themodel during training. The model may then be trained using trainingdata. For this, one can optionally feed in validation data to validateduring training—e.g., evaluate model accuracy with testing datacomparing predictions to ground truth (e.g., DICE coefficients, labelspassed in folder structure metadata, and the like). Modelhyperparameters can be optimized using routines like cross-validation,grid search, and so on, to propose the best parameters leading tohighest accuracy, lowest bias, and/or lowest variance. The model can besaved to be called at runtime, where the model may have an API to becalled programmatically at runtime.

As mentioned above, other training techniques are also or insteadpossible for training one or more of the models described herein. Itwill also be understood that, where a model is described as performingan action or function, this may include a single model or a combinationof a plurality of models.

As shown in step 816, the method 800 may include predicting a healthcharacteristic of an animal from which the biological sample originated.The health characteristic as described herein may be any attributeuseful for sustaining or improving the health and well-being of theanimal, and/or any characteristic that could identify a potential healthissue. For example, as discussed herein, in an aspect where thebiological sample is a stool sample, the health characteristic mayinclude a classification on the Bristol stool scale, stool mass toanimal mass ratio, detection of blood and/or foreign objects in stool,and the like.

As shown in step 818, the method 800 may include analyzing one or morehealth characteristics predicted by the model, i.e., a healthcharacteristic of the animal associated with the biological sample. Forexample, this step 818 may include analyzing a health characteristic inview of a reference database to determine a treatment for the animalassociated with the biological sample. The reference database may be ahistorical database that includes data from analyses of other biologicalsamples—e.g., biological samples from the same animal or biologicalsamples from different animals.

As shown in step 820, the method 800 may include performing a laboratoryanalysis or another analysis of the biological sample separate from theimage analysis described above. This analysis may be used as a factor inpredicting the health characteristic of the animal. Also, or instead,this analysis may be used to test the image analysis techniquesdescribed above, and/or for providing ground truth data. By way ofexample, when the biological sample includes a stool sample, this step820 may include performing microbiome DNA gene sequencing on the stoolsample. To this end, the method 800 may further include applying resultsfrom the microbiome DNA gene sequencing as a factor in predicting thehealth characteristic of the animal. Also, or instead, when thebiological sample includes a stool sample, this step 820 may includeperforming metabolomics sequencing on the stool sample. To this end, themethod 800 may further include applying results from the metabolomicssequencing as a factor in predicting the health characteristic of theanimal.

Step 820 may also or instead include performing further analysis such asone or more of mass spectroscopy, conductance, and rheology, e.g., whenthe biological sample includes a stool sample. Again, results of thisfurther analysis may be used as a factor in predicting the healthcharacteristic of the animal and/or for training a model.

As shown in step 822, the method 800 may include providing a treatmentfor the animal, e.g., in view of the health attribute that was at leastin part predicted by the applied model. The treatment may include one ormore of a food, a liquid, a supplement, a behavior recommendation (e.g.,sleep, exercise, enrichment, and the like), and/or a medicine. Forexample, the treatment may include a personalized dietary supplement forthe animal. Such a personalized dietary supplement may include apredetermined amount of one or more of a probiotic, a prebiotic, adigestive enzyme, an anti-inflammatory, a natural extract, a vitamin, amineral, an amino acid, a short-chain fatty acid, an oil, a formulatingagent, and the like.

Thus, an embodiment executing the method 800, or any of the methodsdisclosed herein, may be used to obtain a personalized dietarysupplement for the animal. A personalized dietary supplement mayinclude, but is not limited to, personalized ingredient levels ofbeneficial ingredients, which can be combined with formulating agentswhere applicable. In this manner, a personalized supplement system maybe provided by the present teachings. Such a personalized supplementsystem may include a durable container structurally configured forstoring the personalized dietary supplement, where the container may begenerally sized and shaped for easy refills. Refill cups may also beprovided, where these may be recyclable and structurally configured foreasy use and fill. A supplement spoon or the like may be used, wheresuch a spoon may be custom-designed to contain a dose that ispersonalized for an animal's specifications. This system can ensurecompliance of dosing a personalized dietary supplement. Also or instead,the personalized supplement system may include a subscription service orthe like, where supplements are mailed or otherwise delivered to anend-user on a predetermined basis and/or in predetermined doses. Forexample, a packet or other container may be used to store a pre-dosedamount of a personalized supplement to be added to the animal's food orwater, where one or more of these packets are supplied to an end user ona predetermined basis.

The treatment may also or instead include a customized health plan forthe animal. The customized health plan may include one or more of abehavioral change, a dietary change, and the like. The customized healthplan may also or instead include a recommendation regarding one or moreof diet, sleep, exercise, an activity, enrichment, a lifestyle change,and the like.

As shown in step 824, the method 800 may include generating andproviding a report for the animal, e.g., where the report includesinformation regarding the health characteristic predicted by the model.

The method 800 described above may be performed using one or morecomputing devices. To that end, the present teachings may include acomputer program product for analyzing a biological sample image toprovide a health assessment of an animal, the computer program productcomprising computer executable code embodied in a non-transitorycomputer readable medium that, when executing on one or more computingdevices, performs the steps of: receiving an image, the image includinga biological sample; identifying and extracting one or more regions ofinterest within the image for further analysis, the one or more regionsof interest including at least a first region of interest having onlythe biological sample therein; calculating one or more of a geometricattribute, a texture attribute, and a color attribute in the firstregion of interest to identify one or more features of the biologicalsample; and applying a model to the one or more features of thebiological sample, the model predicting a health characteristic of ananimal that associated with the biological sample.

FIG. 9 is a flow chart of a method of analyzing an image of a biologicalsample to provide a health assessment of an animal, in accordance with arepresentative embodiment. In general, the method 900 may utilize amodel (e.g., a correlation model), which may be the same or similarmodel trained as per the description of FIG. 6 . As such, the method 900of FIG. 9 may be similar to the technique described with reference toFIG. 7 , where it will be understood that steps of these techniques maybe substituted and/or supplemented for one another.

As shown in step 902, the method 900 may include identifying one or moreassociations between (i) one or more features and/or other dataascertained from an analysis of an image of a biological sample and (ii)one or more attributes and/or characteristics of an animal that isassociated with the biological sample.

Thus, this step 902 may also or instead include identifying one or morefeatures of a biological sample contained in an image. Thus, thefeatures or other data ascertained from an image of a biological samplemay be discovered, uncovered, calculated, and/or identified from one ormore of the various image analysis techniques described herein—e.g.,performed on a digital photograph of the biological sample. By way ofexample, such features may include one or more of an area, acircularity, an eccentricity, a major axis, a minor axis, a perimeter,an equivalent diameter, an extent, a minor-major-axis ratio, solidity,and the like.

The attributes and/or characteristics of an animal that are associatedwith the biological sample may include the presence or absence of asubstance (e.g., one or more of a microbiome characteristic and ametabolome characteristic), a health characteristic, a score (e.g., aBristol stool score), and the like, including combinations thereof. Byway of example, such attributes and/or characteristics may include, orotherwise assist in the identification of, one or more of an indicationof diet (e.g., carbohydrate intake, fresh food intake, protein intake,fat intake, metabolism thereof, and so on), an indication of metabolism,an indication of weight of the animal, an indication of a healthcondition or sickness (e.g., cancer, irritable bowel syndrome (IBS),obesity, and so on), pathogen presence or absence, parasite presence orabsence, and the like. It will be understood that the presence of onesubstance may indicate the presence or absence of another substance, andvice-versa—where such inferences can be readily included in the presentteachings.

By way of example, some possible associations will now be described inthe context of a stool sample—i.e., correlations from image features andhealth-related attributes and/or characteristics such as the presence orabsence of a gastrointestinal microbiome characteristic. The equivalentdiameter of a stool sample can have a correlation to the presence orabsence of one or more of the Bacteroidetes phylum, the Prevotella genus(the presence or absence of which can lead to a further indication ofcarbohydrate load of the animal's diet, which can be helpful for dietaryrecommendations), the Firmicutes phylum, specifically the Ruminococcusgenus (the presence or absence of which can lead to a further indicationof protein intake of the animal, which can be helpful for dietaryrecommendations; also, it is noted that the presence or absence ofFirmicutes may be strongly inversely correlated to the presence orabsence of one or more of Bacteroidetes), among others. In this manner,if the equivalent diameter is relatively large, this can indicate apresence of the Prevotella phylum, which can infer a relativelyhigh-carbohydrate diet and lead to a recommendation of adding moreprotein and reducing carbohydrates in the animal's diet in order tooptimize animal health. The area of a stool sample can have acorrelation to the presence or absence of one or more of theActinobacteria phylum (the presence or absence of which can lead to afurther indication of fat intake of the animal, which can be helpful fordietary recommendations), the Collinsella phylum (the presence orabsence of which can lead to a recommendation for detoxification and/oradditional pathogen testing), among others. The major axis, the minoraxis, and/or the perimeter can have a correlation to the presence orabsence of one or more of the Blautia genus (the presence or absence ofwhich can lead to a recommendation for anti-inflammatory compounds thatcan help with IBS or the like), the Actinobacteria phylum (the presenceor absence of which can lead to a further indication of fat intake ofthe animal, which can be helpful for dietary recommendations), theCollinsella phylum, the Bacteroides genus (the presence or absence ofwhich can lead to a recommendation for prevention of harmful bacteria),among others. The extent and solidity can have a correlation to thepresence or absence of one or more of Clostridium Hiranonis (thepresence or absence of which can lead to a further indication of proteinand/or carbohydrate intake of the animal, which can be helpful fordietary recommendations), the Megamonas genus (the presence or absenceof which can lead to an indication of metabolism and/or weight gain),the Collinsella genus, among others. The minor-major-axis ratio can havea correlation to the presence or absence of one or more of theActinobacteria phylum, the Collinsella genus, among others. Othercorrelations are also or instead possible.

By way of example, further possible correlations will now be describedin the context of a stool sample—i.e., correlations from image featuresand health-related attributes and/or characteristics such as thepresence or absence of metabolome subsystems. The eccentricity can havea correlation to the presence or absence of one or more metabolitesindicating carbohydrate intake, virulence, disease, and defense. Theequivalent diameter can have a correlation to the presence or absence ofone or more metabolome sub-systems indicating carbohydrate intake, aminoacids and derivatives, protein metabolism, cofactors, vitamins,prosthetic groups, pigments, virulence, disease, and defense. Othercorrelations are also or instead possible.

In this manner, the present teachings can be used to validate one ormore of the aforementioned correlations. For instance, using an aspectof the present teachings, the general inverse relationship between theBacteriodetes phylum and the Firmicutes phylum was capturedempirically—which is something that was not thought possible using onlyimage analysis and a trained correlation model.

Metabolome insights may be generated using the same or similartechniques to that can be used to generate microbiome insights. As anexample, metabolome characteristics can be directly correlated tofeatures and/or attributes calculated from an image (e.g., geometry,texture, color, CNN features, and so on). Metabolome characteristics canalso or instead be indirectly correlated (e.g., a second degreeassociation) to image features and/or attributes via a secondaryfeature, such as a microbiome characteristic. Moreover, this process maybe switched—where it is possible for a microbiome characteristic to beindirectly correlated with an image feature using a metabolomecharacteristic.

Microbiome and metabolome characteristics may also or instead becorrelated with a decomposed vector, such as eigenvectors resulting fromPrinciple Component Analysis (PCA). PCA is a common routine used fordimension reduction, which may be valuable when analyzinghigh-dimensionality data in metabolome and microbiome domains. Imageattributes and/or features could be correlated to these compositevectors, and can allow for a prediction of a microbiome or metabolomecharacteristic. The reduced dimensionality eigenvectors may also orinstead be used for clustering (e.g., classification of various types ofresponse).

It will be generally understood that, in the context of this method 900and generally in the techniques described throughout this disclosure,that a “microbiome characteristic” shall generally include a measurableproperty of a component of the microbiome, for example the specificpresence or absence of a taxonomic grouping such as a phylum, group,order, family, genus, and/or species. Similarly, a “metabolomecharacteristic” shall generally include a measurable property of acomponent and/or subsystem of the metabolome, and/or metabolicfunctions. In this manner, associations between features of a biologicalsample calculated from a region of interest of an image of thebiological sample and one or more a microbiome characteristic and ametabolome characteristic may be useful in determining a state of thebiological sample with regard to the microbiome and/or metabolome asexplained in more detail below.

As shown in step 904, the method 900 may include storing theassociations (e.g., the correlations described above) or otherwisemaking them available to a model (e.g., a correlation model). Forexample, associations may be stored in a reference database accessibleto the model.

As shown in step 906, the method 900 may include receiving one or morefeatures of a biological sample (e.g., a stool sample) calculated fromone or more regions of interest within an image including the biologicalsample. The features may relate to at least one of a geometricattribute, a color attribute, a texture attribute, and the like, of thebiological sample. It will be understood that one or more of theseattributes may be derived attributes. For example, where the featuresrelate to a color attribute, such a color attribute may include one ormore of a color and a derived attribute related to the color (e.g.,using k-means color segmentation for calculating color homogeneity,where derived features are created). In a more simplistic example,certain colors can derive a dietary attribute, such as an intake of aparticular food or substance.

As shown in step 908, the method 900 may include applying a model to afeature of the biological sample and the number of associations todetermine useful insight regarding the animal. More specifically, incertain aspects where the biological sample is stool, the method 900 mayinclude applying—using a model created by identifying a number ofcorrelations and/or associations between one or more image-basedfeatures of stool and one or more of a microbiome characteristic and/ora metabolome characteristic in stool—the features of the stool sample tothe number of correlations and/or associations in the model to determinea likelihood of a state of one or more of a microbiome and a metabolomein the stool sample.

It will be understood that the “state” of one or more of a microbiomeand a metabolome in a biological sample (e.g., stool sample) refers to asnapshot in time of a dynamic ecosystem of the biological sample. Forexample, the state of one or more of a microbiome and a metabolome in abiological sample may include the makeup of a particular animal'sgastrointestinal microbiome and/or metabolome, where this makeup can bechanging over time and can be unique from animal to animal. In thismanner, the state of one or more of a microbiome and a metabolome in abiological sample may include various characteristics of one or more ofa microbiome and a metabolome as those characteristics are definedherein. The state of one or more of a microbiome and a metabolome in abiological sample may thus include a presence or absence of a particularmicrobiome and/or particular metabolome in the biological sample, and/oranother characteristic related thereto. The state of one or more of amicrobiome and a metabolome in a biological sample may also or insteadinclude the distribution of a particular microbiome characteristicand/or metabolome characteristic in a biological sample, which caninclude a relative amount and/or number of one or more of a microbiomecharacteristic and a metabolome characteristic that is present. Forexample, this distribution may include a relative abundance of a certainindividual and/or sets of microbiome characteristics and/or metabolomecharacteristics, which may for example be relative to a certainpredetermined threshold number (e.g., an expected average), relative toan abundance of other different microbiome and/or metabolomecharacteristics, relative to an abundance of another substance (e.g.,biological and/or non-biological matter), combinations thereof, and soon. Further, because the method 900 may be performed solely based on animage of a biological sample to make the determination of the state ofone or more of the microbiome and metabolome, this determination may bean estimate and/or hypothesis, since the determination may not be basedon a laboratory analysis. To this end, the method 900 may includedetermining a likelihood of a state of one or more of a microbiome and ametabolome in a biological sample solely based on an image analysis. Themethod 900 can also or instead be utilized in conjunction with alaboratory analysis, e.g., for testing or otherwise.

The method 900 may also or instead include training the model, e.g.,using hundreds of datapoints. After a model 900 has been trained, it maybe loaded to receive inputs such as the image features described herein.

As shown in step 910, the method 900 may include predicting a healthcharacteristic of an animal associated with the biological sample. Forexample, in certain aspects where the biological sample is stool, themethod 900 may include predicting a health characteristic of an animalthat deposited the stool sample based on at least the likelihood of thestate of one or more of the microbiome and the metabolome in the stoolsample, and/or another characteristic related thereto. This maysimilarly be done for non-stool biological samples.

As shown in step 912, the method 900 may include providing a treatment(which may include a recommendation) in view of the healthcharacteristic.

FIG. 10 shows an image and various color planes thereof, in accordancewith a representative embodiment. Specifically, FIG. 10 includes animage 1000 featuring a resting surface 1010 that may be structurallyconfigured for use in image analysis of a biological sample (e.g., astool sample) that is at least partially disposed thereon. To that end,the resting surface 1010 may include one or more predetermined markingsthereon as described herein, e.g., where such markings may be used forextracting one or more of the color attribute and the dimensionalattribute as described herein. By way of example, the figure showsnormalization of example color planes using the predetermined markingsof the resting surface 1010—i.e., the red color plane 1012, the greencolor plane 1014, and the blue color plane 1016.

FIG. 11 shows an image and segmentation thereof, in accordance with arepresentative embodiment. Specifically, FIG. 11 includes an image 1100featuring a biological sample in the form of a stool sample 1102disposed on a resting surface 1104, where the image 1100 furtherincludes a background portion 1106. The figure further shows a firstsegmentation 1110 of the image, which substantially isolates the stoolsample 1102, and a second segmentation 1112 of the image isolates justthe resting surface 1104, which can be used for further processing(e.g., for calculating aspect ratios, for image normalization, and soon).

As described herein, an output of the present teachings may include aBristol stool score, e.g., where an image of a stool sample is analyzedaccording to one or more aspects of the present teachings. That is, inan aspect, trained models may be used to output a prediction, such as aBristol stool score. To that end, a color model and a geometric modelmay each be used to predict a Bristol stool score based on differentfeatures. By way of example, for an arbitrary image, the color model maypredict a Bristol stool score of 4 and the consistency model may predicta Bristol stool score of 3. The overall health index, which may includea system of models, may be calculated as the expected average of theseindividual models, where weights can be added to modify per animal asnecessary. For example, color and consistency may be weighted equally(i.e., x=y=1) so the average is =(4*x+3*y)/2.

As described herein, the present teachings may include featureextraction from an image of a biological sample. From a high level,features may be extracted mathematically from the input images. Forexample, for the geometry attribute(s), the input image may bebinarized, where pixel distances are calculated from various properties.By way of example, for major/minor ratio, the center of the eclipse maybe calculated, and then the distance from the center to thecorresponding point may be calculated. These units may be in pixels,which is why the aspect ratio may be used to convert back to anotherform such as inches or millimeters.

FIG. 12 is a flow chart of a method of formulating a personalizedproduct for an animal. It will be understood that the method 1200,similar to the other methods shown and described herein, may be combinedwith, supplemented to, and/or substituted for any of the othertechniques described herein. In this manner, it will be understood thatthe method 1200 of formulating a personalized product for an animal mayinclude use of an image analysis of a biological sample, e.g., wheresuch an image analysis forms the entire basis for the formulation of thepersonalized product.

The personalized product may include a personalized dietary product. Thepersonalized dietary product may include one or more of a food, asupplement, a medicine, and the like. To this end, the personalizeddietary product may include a predetermined amount of one or more of aprobiotic, a prebiotic, a digestive enzyme, an anti-inflammatory, anatural extract, a vitamin, a mineral, an amino acid, a short-chainfatty acid, an oil, a formulating agent, and the like. The personalizedproduct including a personalized dietary product may be advantageouslycreated based on analysis of a biological sample in the form of a stoolsample.

The personalized product may also or instead include a non-dietaryproduct. For example, the personalized product may include one or moreof a grooming product, a shampoo, a conditioner, a lotion, a cream, amedicine, an ear drop, an eye drop, a topical substance, a toothpaste,an oral rinse, a chew (e.g., a chew toy or the like that can be providedfor health benefits such as oral hygiene), and the like. Thepersonalized product including a non-dietary product may beadvantageously created based on analysis of a biological sample in theform of a non-stool sample, such as an image of a portion of an animal(e.g., fur, ear, mouth, eye, and so on as described herein).

As shown in step 1202, the method 1200 may include receiving an imageincluding a biological sample therein, e.g., a stool sample.

As shown in step 1204, the method 1200 may include applying a model tothe image to extract one or more features of the biological sample. Themodel may be any as described herein, and the extraction of the featuresof the biological sample may utilize any of the image analysistechniques described herein.

As shown in step 1206, the method 1200 may include, based at least onone or more features of the biological sample extracted from the model,selecting one or more ingredients of a personalized product for ananimal from which the biological sample originated. Thus, the method1200 may include creating a unique formula for the personalized productby selecting one or more ingredients and use levels for an animal fromwhich the biological sample originated. In this manner, it will beunderstood that the personalized product may include a personalizedformula.

As shown in step 1208, the method 1200 may include producing thepersonalized product, which may include combining the one or moreingredients (e.g., at custom use levels) to form the personalizedproduct, e.g., to produce a personalized formula. For example, this mayinclude combining all of the ingredients into a single product (e.g.,where the ingredients form one or more of a powder, a pill, a paste, aliquid, a solid, and so on). This may also or instead include combiningthe ingredients into a single container or housing, e.g., for shippingand/or transport, but where individual ingredients may be separated fromone another, which can be advantageous for particular dosing schemes.

As shown in step 1210, the method 1200 may include packaging thepersonalized product, e.g., for distribution such as shipping to a user.Stated otherwise, this may include packaging a personalized formula,e.g., for distribution such as shipping to a user, thereby comprising apersonalized product.

As shown in step 1212, the method 1200 may include distributing thepersonalized product to one or more of the animal and a user associatedwith the animal (e.g., an owner of the animal, a veterinarian, aphysician, and so on).

As shown in step 1214, the method 1200 may include dosing thepersonalized product for the animal. This may include separatingparticular doses of the personalized product for ease of use by anend-user. This may also or instead include providing a dosing componentsuch as a scoop, a spoon, a vial, a measuring cup, a syringe, a packet(e.g., a blister pack, foil, etc.), and so on, in conjunction withproviding the personalized product. The dosing method may also orinstead include a personalized dosing component to assist in ensuringaccurate administration of the personalized product. It will beunderstood that the personalized dosing component may be custom-tailoredfor a specific animal or set of animals.

Several other aspects of the present teachings are described below byway of example.

Data Wrangling and Preparation of Data

Stool Health Features:

Data may be extracted from a source, such as a spreadsheet and/or arelational/non-relational database stored locally or on the cloud. Datamay be loaded into software, such as a high-level, general-purposeprogramming language. Data may be transformed as needed (e.g., scalingdata to unit mean and variance). Data can continue to be cleaned priorto analysis (e.g., string formatting, removing null values, convertingobject types). Feature matrix X may be formatted and vector y may belabeled. Feature matrix X can have any number of attributes and/orfeatures, including geometric attributes, color attributes, and so on.Feature vector y may contain a true class label such as Bristol stoolscore.

Semantic Segmentation:

Images and corresponding ground truth labels may be stored in a datasource (e.g., local and/or cloud). Images may be extracted from a source(e.g., local and/or cloud based storage). Images may be resized tostandard or otherwise predetermined sizes. Images may be augmented(e.g., cropping, flipping vertically/horizontally, shearing, rotating,etc.) to increase dataset size to both raw and ground truth. It will beunderstood that these steps may be automated with software.

Modeling

Stool Health Features:

The dataset be split into training, testing, and/or validation sets,using a portion of data (e.g., about 70%-80%) to train the model, wheresome data (e.g., about 20%-30%) is reserved to test the data aftertraining, and for validating (e.g., about 10%) the model duringtraining. The model may then be created, e.g., from loading from apackage or a defined architecture. The model may be trained usingtraining data, where validation data is optionally fed-in to validateduring training. Model accuracy may be evaluated with testing data,e.g., comparing predictions to ground truth. Model hyperparameters canbe optimized using routines such as cross-validation, grid search, andthe like to propose parameters that can lead to desired accuracy, bias,and/or variance. The model may be saved to be called at runtime, wherethe model may have an API to be called programmatically at runtime. Itwill be understood that these steps may be automated with software.

Semantic Segmentation:

A custom convolutional neural network (CNN) architecture or instantiateCNN architecture that is known (e.g., U-Net) may be defined. The datasetbe split into training, testing, and/or validation sets, using a portionof data (e.g., about 70%-80%) to train the model, where some data (e.g.,about 20%-30%) is reserved to test the data after training, and forvalidating (e.g., about 10%) the model during training. The model may betrained using training data, where validation data is optionally fed-into validate during training. Model accuracy may be evaluated withtesting data, e.g., comparing predictions to ground truth (e.g., DICEcoefficients). Model hyperparameters can be optimized using routinessuch as cross-validation, grid search, and the like to proposeparameters that can lead to desired accuracy, bias, and/or variance. Themodel may be saved to be called at runtime, where the model may have anAPI to be called programmatically at runtime.

Code Example

It will be understood that particular portions of the disclosedtechniques for analyzing an image of a biological sample image toprovide a health assessment of an animal associated therewith mayutilize preexisting algorithms and/or models that can be adapted toperform the particular portions. By way of example, feature engineeringwill now be discussed in this context. For performing featureengineering of a stool sample image according to an aspect, k-mean colorsegmentation may involve pulling the top ten contiguous color regionsfrom the image and calculating the percent abundance (e.g., color plane1 may take 30% of the total area of a stool). To create these featurevectors, planes may be counted that occur at more than about 10% andabout 20% abundance. First, pandas may be imported as pd and an aspectof the present teachings can count the number of planes above a relativeabundance using a group by method:

import pandas as pd df = pd.read_csv(‘hex_codes_jpg_11june2020.csv’)over10 = df.loc[df.Rel_Pct>0.1,:].groupby(“Filename”).count( )[‘index’]over 20 = df.loc[df.Rel_Pct>0.2,:].groupby(“Filename”).count( )[‘index’]# cast these back into original filenames so it can merge back onto theoriginal dataframe

The following is an example for code for color homogeneity featureengineering:

unique_files = list(df.Filename.unique( )) over10_fv =np.zeros((len(unique_files))) over20_fv = np.zeros((len(unique_files)))for i in range(0,len(unique_files)): file = unique_files[i] if file inover10.index: over10_fv[i] = over10[file] if file in over20.index:over20_fv[i] = over20[file] color_features =pd.DataFrame({‘NumOver10’:over10_fv,‘NumOver20’:over20_fv},index=df.Filename.unique( )) # color homogeneity

The following is a machine learning example that can be used inconsistency models, which uses a transformation of the data (an aspectof feature engineering). First, the appropriate libraries can beimported and the data may be split into testing and training data setsusing the sklearn train_test_split module:

# import data import pandas as pd import matplotlib.pyplot as plt importseaborn import numpy as np df =pd.read_excel(“../Data/geometries_all.xlsx”) fromsklearn.model_selection import train_test_split x =df[[“Area”,”Circularity”,”Eccentricity”, “Major Axis”, “Minor Axis”,“Perimeter”]].values y = df[“BSS”].values x_train, x_test, y_train,y_test = train_test_split(x,y,test_size=0.2,random_state=0)

The machine learning model can then be developed/deployed. For example,the following shows example code for training a linear support vectormachine (SVM) classifier and making predictions with the machinelearning model:

from sklearn.svm import SVC from sklearn.preprocessing importStandardScaler scaler = StandardScaler( ) x_scaler = scaler.fit(x_train)x_train_scaled = x_scaler.transform(x_train) x_test_scaled =x_scaler.transform(x_test) svm_model_linear = SVC(kernel = ‘linear’, C =1).fit(x_train, y_train) svm_predictions =svm_model_linear.predict(x_test)

The following shows example code for calculating model accuracy:

-   -   accuracy=svm_model_linear.score(x_test_scaled, y_test)

It will be understood that the code recited herein is provided by way ofexample and understanding only, and that other similar code may be used.Also, it would be apparent to one skilled in the art how to practice thepresent teachings even without such examples of code, e.g., because ofthe ample disclosure provided herein regarding the functionality of themodels and the like described herein. One skilled in the art willfurther understand that code examples provided above reference readingfiles from a disk, e.g., raw text files, and given the significantdetail of the present teachings described herein, such files and codecould be created by a skilled artisan to recreate aspects of the presentteachings.

The above systems, devices, methods, processes, and the like may berealized in hardware, software, or any combination of these suitable fora particular application. The hardware may include a general-purposecomputer and/or dedicated computing device. This includes realization inone or more microprocessors, microcontrollers, embeddedmicrocontrollers, programmable digital signal processors or otherprogrammable devices or processing circuitry, along with internal and/orexternal memory. This may also, or instead, include one or moreapplication specific integrated circuits, programmable gate arrays,programmable array logic components, or any other device or devices thatmay be configured to process electronic signals. It will further beappreciated that a realization of the processes or devices describedabove may include computer-executable code created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software. In another aspect, themethods may be embodied in systems that perform the steps thereof, andmay be distributed across devices in a number of ways. At the same time,processing may be distributed across devices such as the various systemsdescribed above, or all of the functionality may be integrated into adedicated, standalone device or other hardware. In another aspect, meansfor performing the steps associated with the processes described abovemay include any of the hardware and/or software described above. Allsuch permutations and combinations are intended to fall within the scopeof the present disclosure.

Embodiments disclosed herein may include computer program productscomprising computer-executable code or computer-usable code that, whenexecuting on one or more computing devices, performs any and/or all ofthe steps thereof. The code may be stored in a non-transitory fashion ina computer memory, which may be a memory from which the program executes(such as random-access memory associated with a processor), or a storagedevice such as a disk drive, flash memory or any other optical,electromagnetic, magnetic, infrared, or other device or combination ofdevices. In another aspect, any of the systems and methods describedabove may be embodied in any suitable transmission or propagation mediumcarrying computer-executable code and/or any inputs or outputs fromsame.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the disclosure to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings.

Unless the context clearly requires otherwise, throughout thedescription, the words “comprise,” “comprising,” “include,” “including,”and the like are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in a sense of “including,but not limited to.” Additionally, the words “herein,” “hereunder,”“above,” “below,” and words of similar import refer to this applicationas a whole and not to any particular portions of this application.

It will be appreciated that the devices, systems, and methods describedabove are set forth by way of example and not of limitation. Absent anexplicit indication to the contrary, the disclosed steps may bemodified, supplemented, omitted, and/or re-ordered without departingfrom the scope of this disclosure. Numerous variations, additions,omissions, and other modifications will be apparent to one of ordinaryskill in the art. In addition, the order or presentation of method stepsin the description and drawings above is not intended to require thisorder of performing the recited steps unless a particular order isexpressly required or otherwise clear from the context.

The method steps of the implementations described herein are intended toinclude any suitable method of causing such method steps to beperformed, consistent with the patentability of the following claims,unless a different meaning is expressly provided or otherwise clear fromthe context. So, for example performing the step of X includes anysuitable method for causing another party such as a remote user, aremote processing resource (e.g., a server or cloud computer) or amachine to perform the step of X. Similarly, performing steps X, Y and Zmay include any method of directing or controlling any combination ofsuch other individuals or resources to perform steps X, Y and Z toobtain the benefit of such steps. Thus, method steps of theimplementations described herein are intended to include any suitablemethod of causing one or more other parties or entities to perform thesteps, consistent with the patentability of the following claims, unlessa different meaning is expressly provided or otherwise clear from thecontext. Such parties or entities need not be under the direction orcontrol of any other party or entity, and need not be located within aparticular jurisdiction.

It should further be appreciated that the methods above are provided byway of example. Absent an explicit indication to the contrary, thedisclosed steps may be modified, supplemented, omitted, and/orre-ordered without departing from the scope of this disclosure.

It will be appreciated that the methods and systems described above areset forth by way of example and not of limitation. Numerous variations,additions, omissions, and other modifications will be apparent to one ofordinary skill in the art. In addition, the order or presentation ofmethod steps in the description and drawings above is not intended torequire this order of performing the recited steps unless a particularorder is expressly required or otherwise clear from the context. Thus,while particular embodiments have been shown and described, it will beapparent to those skilled in the art that various changes andmodifications in form and details may be made therein without departingfrom the spirit and scope of this disclosure and are intended to form apart of the invention as defined by the following claims, which are tobe interpreted in the broadest sense allowable by law.

What is claimed is:
 1. A method of analyzing a stool sample image toprovide a health assessment of an animal, the method comprising:receiving an image of a stool sample from a camera of a smartphone;identifying and extracting one or more regions of interest within theimage for analysis, wherein a microbiome present within the stool sampleis invisible within the one or more regions of interest of the image;identifying at least one of a geometric attribute, a color attribute,and a texture attribute of the stool sample, and determining one or morefeatures of the stool sample based thereon; applying, using a modelcreated by identifying a number of associations between one or moreimage-based features of stool and a microbiome characteristic in stool,the one or more features of the stool sample to the number ofassociations in the model to determine a likelihood of a state of amicrobiome in the stool sample, the state including a distribution of aparticular microbiome characteristic, the distribution including anabundance of the microbiome relative to at least one of (i) apredetermined threshold abundance and (ii) an amount of anothersubstance in the stool sample; based on at least the likelihood of thestate of the microbiome, predicting a health characteristic of an animalthat deposited the stool sample; and providing a treatment in view ofthe health characteristic.
 2. The method of claim 1, wherein thetreatment includes a customized health plan for the animal.
 3. Themethod of claim 2, wherein the customized health plan includes one ormore of a behavioral change and a dietary change.
 4. The method of claim2, wherein the customized health plan includes a recommendationregarding one or more of diet, sleep, exercise, and an activity.
 5. Themethod of claim 1, wherein the treatment includes one or more of a food,a supplement, and a medicine.
 6. The method of claim 1, wherein thetreatment includes a personalized dietary supplement for the animal. 7.The method of claim 6, wherein the personalized dietary supplementincludes a predetermined amount of one or more of a probiotic, aprebiotic, a digestive enzyme, an anti-inflammatory, a natural extract,a vitamin, a mineral, an amino acid, a short-chain fatty acid, an oil,and a formulating agent.
 8. The method of claim 1, wherein the one ormore features relate to the geometric attribute, the color attribute,and the texture attribute of the stool sample, and wherein the geometricattribute includes one or more of a geometric property and a derivedattribute related to geometry, the color attribute includes one or moreof a color and a derived attribute related to the color, and the textureattribute includes one or more of a texture property and a derivedattribute related to texture.
 9. The method of claim 1, wherein the oneor more features are identified using a convolutional neural network(CNN) model.
 10. The method of claim 1, further comprising providing areport for the animal that includes the health characteristic.
 11. Themethod of claim 1, wherein the one or more features of the stool sampleinclude at least one of a color, a texture, a number of binaries, anarea, a perimeter, a circularity, a mass, an eccentricity, a major axis,a minor axis, a viscosity, a consistency, a moisture content, asolidity, an extent, an equivalent diameter, a specularity, a coherence,a reflectance, a diffusivity, and a presence of a non-stool substance.12. The method of claim 11, wherein the one or more features of thestool sample include the mass, and wherein the mass is calculated from ageometry and the texture attribute of the stool sample.
 13. The methodof claim 11, wherein the one of more features of the stool sampleinclude the mass, and wherein the mass is calculated from at least oneof a color and a derived color vector of the stool sample.
 14. Themethod of claim 11, wherein the health characteristic includes a Bristolstool score.
 15. The method of claim 1, wherein the image including thestool sample includes a resting surface having markings thereon, themarkings including one or more of a known size, a known shape, and aknown color, wherein the markings are used at least in part fordetermining the one or more features of the stool sample.
 16. The methodof claim 1, further comprising receiving metadata associated with theimage, the metadata including a questionnaire response related to one ormore of a health, a behavior, a current diet, a supplement, amedication, ethnographic information, a breed, a weight of the animal, aweight of the stool sample, and a size of the animal, and wherein themetadata is used at least in part in predicting the healthcharacteristic.
 17. The method of claim 1, wherein the model is trainedto identify a number of associations between one or more image-basedfeatures of stool and a metabolome characteristic in stool, the methodfurther comprising applying the model to the one or more features of thestool sample to determine a likelihood of a state of a metabolome in thestool sample, the state including at least one of (i) a presence orabsence of a particular metabolome, and (ii) a distribution of aparticular metabolome characteristic.
 18. A computer program product foranalyzing a stool sample image to provide a health assessment of ananimal, the computer program product comprising computer executable codeembodied in a non-transitory computer readable medium that, whenexecuting on one or more computing devices, performs the steps of:receiving an image of a stool sample from a camera of a smartphone;identifying and extracting one or more regions of interest within theimage for analysis, wherein a microbiome present within the stool sampleis invisible within the one or more regions of interest of the image;identifying at least one of a geometric attribute, a color attribute,and a texture attribute of the stool sample, and determining one or morefeatures of the stool sample based thereon; applying, using a modelcreated by identifying a number of associations between one or moreimage-based features of stool and a microbiome characteristic in stool,the one or more features of the stool sample to the number ofassociations in the model to determine a likelihood of a state of amicrobiome in the stool sample, the state including a distribution of aparticular microbiome characteristic, the distribution including anabundance of the microbiome relative to at least one of (i) apredetermined threshold abundance and (ii) an amount of anothersubstance in the stool sample; based on at least the likelihood of thestate of the microbiome, predicting a health characteristic of an animalthat deposited the stool sample; and providing a treatment in view ofthe health characteristic.
 19. The computer program product of claim 18,wherein the model is trained to identify a number of associationsbetween one or more image-based features of stool and a metabolomecharacteristic in stool, and wherein the computer executable code, whenexecuting in one or more computing devices, further performs the step ofapplying the model to the one or more features of the stool sample todetermine a likelihood of a state of a metabolome in the stool sample,the state including at least one of (i) a presence or absence of aparticular metabolome, and (ii) a distribution of a particularmetabolome characteristic.
 20. A system for analyzing a stool sampleimage to provide a health assessment of an animal, the systemcomprising: a data network; a smartphone coupled to the data network,the smartphone including a camera; and a remote computing resourcecoupled to the data network and accessible to the smartphone through thedata network, the remote computing resource including a processor and amemory, the memory storing code executable by the processor to performthe steps of: receiving an image, taken by the camera, from thesmartphone over the data network, the image including a stool sample;identifying and extracting one or more regions of interest within theimage for analysis, wherein one or more of a microbiome and a metabolomepresent within the stool sample is invisible within the one or moreregions of interest of the image; identifying at least one of ageometric attribute, a color attribute, and a texture attribute of thestool sample, and determining one or more features of the stool samplebased thereon; applying, using a model created by identifying a numberof associations between one or more image-based features of stool andone or more of a microbiome characteristic and a metabolomecharacteristic in stool, the one or more features of the stool sample tothe number of associations in the model to determine a likelihood of astate of one or more of a microbiome and a metabolome in the stoolsample, the state including a distribution of a particular microbiomecharacteristic and/or a particular metabolome characteristic, thedistribution including an abundance of the microbiome relative to atleast one of (i) a predetermined threshold abundance and (ii) an amountof another substance in the stool sample; based on at least thelikelihood of the state of one or more of the microbiome and themetabolome, predicting a health characteristic of an animal thatdeposited the stool sample; and transmitting a treatment to thesmartphone over the data network in view of the health characteristic.